4 ms·

Servers, even ones that run NTP, aren't always on time. I had one the other day that somehow got 30 minutes behind.

existencebox 11y ago
It's the more subtle instances of this that always are the most worrying to me, to be honest. Imagine you have a data pipeline that spans multiple hosts, and you want telemetry on when an entity passes through each step? Suddenly you have to start caring about clock alignment in an order of magnitude comparable with network transit/computation time (microseconds) and that level of skew happens VERY often. There are certainly ways around this in how you build your system, but it's something that _must_ be considered and I find often isn't.
- toomuchtodo 11y ago
  If you're monitoring other system metrics, and your application is time-dependent, you should be monitoring system time and ntpd stats as well. /u/devnull42 is doing it right: https://news.ycombinator.com/item?id=11075883 https://news.ycombinator.com/item?id=11075883
  - existencebox 11y ago
    For critical time dependent systems, I absolutely agree. I was unclear in that I was referring to far more mundane issues where slight skew can be problematic. I've seen telemetry systems in which the aformentioned deltas would be computed by doing a timedelta() on the recorded timestamp from each machine. With even subsecond skew, you can get some really interesting artifacts; negative durations, that sort of thing.
devnull42 11y ago
I run some pretty large NTP deployments and our NTP drift is never really over 1 second. Now the difference between NTP serverside drift and client server-side "skew" isn’t something I have looks at but for the most part I get a nagios alert on anything over 1 second of drift with 5 seconds being the most I have ever seen.
- newman314 11y ago
  Can you share more detail as to how you would alert on drift? What acts as the canonical source?
  - simcop2387 11y ago
    Not sure how you'd configure it, but you don't actually need a canonical source. You just compare it against all/many other servers and look at the difference. If there's any real skew you need to look at all of them that had a difference then but you don't need a canonical source anymore.

It's the more subtle instances of this that always are the most worrying to me, to be honest. Imagine you have a data pipeline that spans multiple hosts, and you want telemetry on when an entity passes through each step? Suddenly you have to start caring about clock alignment in an order of magnitude comparable with network transit/computation time (microseconds) and that level of skew happens VERY often. There are certainly ways around this in how you build your system, but it's something that _must_ be considered and I find often isn't.

toomuchtodo 11y ago
If you're monitoring other system metrics, and your application is time-dependent, you should be monitoring system time and ntpd stats as well. /u/devnull42 is doing it right: https://news.ycombinator.com/item?id=11075883 https://news.ycombinator.com/item?id=11075883
- existencebox 11y ago
  For critical time dependent systems, I absolutely agree. I was unclear in that I was referring to far more mundane issues where slight skew can be problematic. I've seen telemetry systems in which the aformentioned deltas would be computed by doing a timedelta() on the recorded timestamp from each machine. With even subsecond skew, you can get some really interesting artifacts; negative durations, that sort of thing.

devnull42 11y ago

I run some pretty large NTP deployments and our NTP drift is never really over 1 second. Now the difference between NTP serverside drift and client server-side "skew" isn’t something I have looks at but for the most part I get a nagios alert on anything over 1 second of drift with 5 seconds being the most I have ever seen.

newman314 11y ago
Can you share more detail as to how you would alert on drift? What acts as the canonical source?
- simcop2387 11y ago
  Not sure how you'd configure it, but you don't actually need a canonical source. You just compare it against all/many other servers and look at the difference. If there's any real skew you need to look at all of them that had a difference then but you don't need a canonical source anymore.