6 ms·

Clock Skew Exists

42 points by charlieegan3 11y ago

danielhlockard 11y ago
Servers, even ones that run NTP, aren't always on time. I had one the other day that somehow got 30 minutes behind.
- existencebox 11y ago
  It's the more subtle instances of this that always are the most worrying to me, to be honest. Imagine you have a data pipeline that spans multiple hosts, and you want telemetry on when an entity passes through each step? Suddenly you have to start caring about clock alignment in an order of magnitude comparable with network transit/computation time (microseconds) and that level of skew happens VERY often. There are certainly ways around this in how you build your system, but it's something that _must_ be considered and I find often isn't.
  - toomuchtodo 11y ago
    If you're monitoring other system metrics, and your application is time-dependent, you should be monitoring system time and ntpd stats as well. /u/devnull42 is doing it right: https://news.ycombinator.com/item?id=11075883 https://news.ycombinator.com/item?id=11075883
    - existencebox 11y ago
      For critical time dependent systems, I absolutely agree. I was unclear in that I was referring to far more mundane issues where slight skew can be problematic. I've seen telemetry systems in which the aformentioned deltas would be computed by doing a timedelta() on the recorded timestamp from each machine. With even subsecond skew, you can get some really interesting artifacts; negative durations, that sort of thing.
- devnull42 11y ago
  I run some pretty large NTP deployments and our NTP drift is never really over 1 second. Now the difference between NTP serverside drift and client server-side "skew" isn’t something I have looks at but for the most part I get a nagios alert on anything over 1 second of drift with 5 seconds being the most I have ever seen.
  - newman314 11y ago
    Can you share more detail as to how you would alert on drift? What acts as the canonical source?
    - simcop2387 11y ago
      Not sure how you'd configure it, but you don't actually need a canonical source. You just compare it against all/many other servers and look at the difference. If there's any real skew you need to look at all of them that had a difference then but you don't need a canonical source anymore.
antirez 11y ago
I posted this comment in Julia's blog 5 hours ago, currently pending. I'm not sure if it will be accepted, but I'm reporting it here: Hello Julia, the kind of skew that is a problem for Redlock, or other algorithms that would rely on the same clock assumptions, is not when the computer "logical" clocks skew, for example by getting a wrong time from NTP. By using the monotonically increasing clock API we only need to care about failures that cause the clock to go faster / slower even if never modified. Also note that there is no issues with clocks of different processes to be at a totally different absolute times, as long as they continue to count the time approximately at the same speed. If you restrict the failures to this class of issues, then to see it in the practice becomes not impossible but extremely rare. Most real world systems out there using proven algorithms, have to do the same assumptions on the fact that the system works in a mostly expected way. For example disk corruptions or network packets corruptions can cause the same issues, even if they are rarely checksummed at application level, or checked with a checksum length that does not make collisions impossible in the long run. Btw kudos for trying to reason with your head instead of trusting other people's or my words.
- alblue 11y ago
  "If you restrict the failures to this class of issues" is exactly where the assumption that clocks behave is hiding in this paragraph. It's a bit like saying "as long as you ignore false, all booleans can be assumed to be true".
  - antirez 11y ago
    If you use the monotonic time API the issues are restricted to this subclass of hardware failures. All the others will be irrelevant, like ntpd setting a random time or alike.
    - querulous 11y ago
      how does using the monotonic api protect against time discontinuity as seen when a vm is migrated from one hypervisor to another for example?
packetized 11y ago
Clock skew is a bad enough thing that it's accepted wisdom to not run timing-sensitive authentication authorities (read: Kerberos, mostly) in a hypervisor or VM container.
- josh2600 11y ago
  This has a few of reasons: 1) all abstraction necessarily adds a performance penalty. 2) Hypervisors or VMs have core affinity, but, often, multiple VMs and Hypervisors are pinned to the same core. Since one cpu cannot expose time to two different vm requests at precisely the same time, there is always some variance. 3) Process isolation is pretty good, but total performance isolation is not quite the same. There are a TON of papers written about VM and hypervisor time management. It's all fascinating and hard.
  - packetized 11y ago
    Absolutely. I'm just surprised that this many people seem to be in the dark about clock synchronicity (or lack thereof) effects on distributed systems.

Clock Skew Exists

42 points by charlieegan3 11y ago

danielhlockard 11y ago

Servers, even ones that run NTP, aren't always on time. I had one the other day that somehow got 30 minutes behind.

existencebox 11y ago
It's the more subtle instances of this that always are the most worrying to me, to be honest. Imagine you have a data pipeline that spans multiple hosts, and you want telemetry on when an entity passes through each step? Suddenly you have to start caring about clock alignment in an order of magnitude comparable with network transit/computation time (microseconds) and that level of skew happens VERY often. There are certainly ways around this in how you build your system, but it's something that _must_ be considered and I find often isn't.
- toomuchtodo 11y ago
  If you're monitoring other system metrics, and your application is time-dependent, you should be monitoring system time and ntpd stats as well. /u/devnull42 is doing it right: https://news.ycombinator.com/item?id=11075883 https://news.ycombinator.com/item?id=11075883
  - existencebox 11y ago
    For critical time dependent systems, I absolutely agree. I was unclear in that I was referring to far more mundane issues where slight skew can be problematic. I've seen telemetry systems in which the aformentioned deltas would be computed by doing a timedelta() on the recorded timestamp from each machine. With even subsecond skew, you can get some really interesting artifacts; negative durations, that sort of thing.
devnull42 11y ago
I run some pretty large NTP deployments and our NTP drift is never really over 1 second. Now the difference between NTP serverside drift and client server-side "skew" isn’t something I have looks at but for the most part I get a nagios alert on anything over 1 second of drift with 5 seconds being the most I have ever seen.
- newman314 11y ago
  Can you share more detail as to how you would alert on drift? What acts as the canonical source?
  - simcop2387 11y ago
    Not sure how you'd configure it, but you don't actually need a canonical source. You just compare it against all/many other servers and look at the difference. If there's any real skew you need to look at all of them that had a difference then but you don't need a canonical source anymore.

antirez 11y ago

I posted this comment in Julia's blog 5 hours ago, currently pending. I'm not sure if it will be accepted, but I'm reporting it here: Hello Julia, the kind of skew that is a problem for Redlock, or other algorithms that would rely on the same clock assumptions, is not when the computer "logical" clocks skew, for example by getting a wrong time from NTP. By using the monotonically increasing clock API we only need to care about failures that cause the clock to go faster / slower even if never modified. Also note that there is no issues with clocks of different processes to be at a totally different absolute times, as long as they continue to count the time approximately at the same speed. If you restrict the failures to this class of issues, then to see it in the practice becomes not impossible but extremely rare. Most real world systems out there using proven algorithms, have to do the same assumptions on the fact that the system works in a mostly expected way. For example disk corruptions or network packets corruptions can cause the same issues, even if they are rarely checksummed at application level, or checked with a checksum length that does not make collisions impossible in the long run. Btw kudos for trying to reason with your head instead of trusting other people's or my words.

alblue 11y ago
"If you restrict the failures to this class of issues" is exactly where the assumption that clocks behave is hiding in this paragraph. It's a bit like saying "as long as you ignore false, all booleans can be assumed to be true".
- antirez 11y ago
  If you use the monotonic time API the issues are restricted to this subclass of hardware failures. All the others will be irrelevant, like ntpd setting a random time or alike.
  - querulous 11y ago
    how does using the monotonic api protect against time discontinuity as seen when a vm is migrated from one hypervisor to another for example?

packetized 11y ago

Clock skew is a bad enough thing that it's accepted wisdom to not run timing-sensitive authentication authorities (read: Kerberos, mostly) in a hypervisor or VM container.

josh2600 11y ago
This has a few of reasons: 1) all abstraction necessarily adds a performance penalty. 2) Hypervisors or VMs have core affinity, but, often, multiple VMs and Hypervisors are pinned to the same core. Since one cpu cannot expose time to two different vm requests at precisely the same time, there is always some variance. 3) Process isolation is pretty good, but total performance isolation is not quite the same. There are a TON of papers written about VM and hypervisor time management. It's all fascinating and hard.
- packetized 11y ago
  Absolutely. I'm just surprised that this many people seem to be in the dark about clock synchronicity (or lack thereof) effects on distributed systems.