4 ms·

Does NTP do that, though? I was under the impression it made time slightly faster/slower until the point is met, if the difference is small (which it will alway

by tiagod 6y ago

Does NTP do that, though? I was under the impression it made time slightly faster/slower until the point is met, if the difference is small (which it will always be on production servers, in principle)

throwaway09223 6y ago
ntpd does indeed skip time, contrary to popular belief. I didn't believe it myself until one day I studied the documentation and code. Check out the "-x" flag in the manpage. I recommend dumping ntpd and using chrony instead, and also configuring chrony for a time smear rather than the leap second as required by the standard.
- segfaultbuserr 6y ago
  > ntpd does indeed skip time The documentation also indicates it's rare for a 7x24 running server to suddenly step time (it only occurs when there's a 128 ms difference), unless you get serious network congestion that prevents sychroninization for an extended period, so a time step in a production server is uncommon. But a standard-compliant negative leap second requires time stepping, it's why I said there might be disruption.
  - deleted 6y ago
    [deleted]
  - throwaway09223 6y ago
    Yes, but the documentation is wrong in its assumption. There are many reasons time may skew >128ms beyond "network congestion." I've dealt with datacenters getting a bad ntp config push, for example, or a bad firewall ruleset which blocks NTP. Fixing the config may inadvertantly step time. There's a whole host of systems issues, sometimes involving VMs and sometimes bare metal which can cause this type of problem. The most simple of which is ntpd not running properly for a while, but this can also happen when systems get overloaded - extreme swapping and the like. ntpd's behavior of non-monotonic time adjustments was brought to my attention after a developer uncovered database corruption, caused by a database which (correctly) assumed that time should be monotonically adjusted while the database server was running. I believe it mis-ordered a transaction log or something along those lines. There are other, better ntpd implementations like chrony which can be configured to never step time. If time needs to be stepped, applications should be stopped and the machine should be removed from service first.
    - segfaultbuserr 6y ago
      Thanks for sharing the experience. > There are other, better ntpd implementations like chrony which can be configured to never step time. I totally agree that ntpd is worse, it has an aging code base with many potential security issues, chrony is a better and cleaner implementation. Nevertheless, I refuse to refer to "ntpd" as a worse implementation because it implemented the leap-second as the standard specified. Also, ntpd can be manually configured to never slew time (excluding leap second) if -x and -g are used. > ntpd's behavior of non-monotonic time adjustments was brought to my attention after a developer uncovered database corruption. BTW, my original topic here is how a negative leap second may cause service disruption, ntpd's behavior is only a footnote to that discussion. I think your experience only strengthened my original argument: if the standard leap second implementation (instead of smearing) is used, which is the case in a default configuration, something might happen. But I'm not sure, a negative leap second doesn't break the monotonic assumption. So it's all speculations...
deleted 6y ago
[deleted]

by tiagod 6y ago

throwaway09223 6y ago

ntpd does indeed skip time, contrary to popular belief. I didn't believe it myself until one day I studied the documentation and code. Check out the "-x" flag in the manpage. I recommend dumping ntpd and using chrony instead, and also configuring chrony for a time smear rather than the leap second as required by the standard.

segfaultbuserr 6y ago
> ntpd does indeed skip time The documentation also indicates it's rare for a 7x24 running server to suddenly step time (it only occurs when there's a 128 ms difference), unless you get serious network congestion that prevents sychroninization for an extended period, so a time step in a production server is uncommon. But a standard-compliant negative leap second requires time stepping, it's why I said there might be disruption.
- deleted 6y ago
  [deleted]
- throwaway09223 6y ago
  Yes, but the documentation is wrong in its assumption. There are many reasons time may skew >128ms beyond "network congestion." I've dealt with datacenters getting a bad ntp config push, for example, or a bad firewall ruleset which blocks NTP. Fixing the config may inadvertantly step time. There's a whole host of systems issues, sometimes involving VMs and sometimes bare metal which can cause this type of problem. The most simple of which is ntpd not running properly for a while, but this can also happen when systems get overloaded - extreme swapping and the like. ntpd's behavior of non-monotonic time adjustments was brought to my attention after a developer uncovered database corruption, caused by a database which (correctly) assumed that time should be monotonically adjusted while the database server was running. I believe it mis-ordered a transaction log or something along those lines. There are other, better ntpd implementations like chrony which can be configured to never step time. If time needs to be stepped, applications should be stopped and the machine should be removed from service first.
  - segfaultbuserr 6y ago
    Thanks for sharing the experience. > There are other, better ntpd implementations like chrony which can be configured to never step time. I totally agree that ntpd is worse, it has an aging code base with many potential security issues, chrony is a better and cleaner implementation. Nevertheless, I refuse to refer to "ntpd" as a worse implementation because it implemented the leap-second as the standard specified. Also, ntpd can be manually configured to never slew time (excluding leap second) if -x and -g are used. > ntpd's behavior of non-monotonic time adjustments was brought to my attention after a developer uncovered database corruption. BTW, my original topic here is how a negative leap second may cause service disruption, ntpd's behavior is only a footnote to that discussion. I think your experience only strengthened my original argument: if the standard leap second implementation (instead of smearing) is used, which is the case in a default configuration, something might happen. But I'm not sure, a negative leap second doesn't break the monotonic assumption. So it's all speculations...

deleted 6y ago

[deleted]