5 ms·

> I don't think a negative leap second would cause much of an issue at all. But nobody has tested it in the wild. It's entirely possible to trigger an existing

> I don't think a negative leap second would cause much of an issue at all. But nobody has tested it in the wild. It's entirely possible to trigger an existing bug in the code base. I expect some disruption from the first negative leap second event. After that everything would probably be gradually stabilized.

jedberg 6y ago
It's constantly tested in the wild. Every time your computer does an NTP update and has to jump ahead one second or more, you've just tested it.
- segfaultbuserr 6y ago
  > Every time your computer does an NTP update and has to jump ahead one second PCs don't run anything crucial, usually you can set your clock to 2001 without visible problems, the same is not true for a server. A server uses ntpd, not a bruce-force ntpdate. Under ordinary conditions, ntpd adjusts the clock in small steps so that the timescale is effectively continuous and without discontinuities. But I think a negative leap second is quite violent.
  - jedberg 6y ago
    I was talking about ntpd. It would do its update after the negative leap second and then think to itself "oh, I'm more than one second behind now, I better start fuzzing and catch up".
    - segfaultbuserr 6y ago
      It's rare for ntpd to step time forward, unless ntpd lost synchronization and the local clock has drifted beyond +/- 128 ms by default. On a 7x24 production server, it's very unlikely, time is always slewed. But a standard leap second implementation requires the addition or deletion of time via adjtimex()'s TIME_INS and TIME_DEL, it's not adjusted gradually. > "oh, I'm more than one second behind now, I better start fuzzing and catch up". No, ntpd would say, oh, leap second, now it's the time to tell kernel to delete a second! And if the kernel's negative leap second code path has a bug, kernel panics and everything crashes. ...Yes, you probably can force ntpd to ignore the leap second deletion and also force NTP to slew time only. This approach is similar to leap second smearing (which is considered a violation of standard but nevertheless an useful hack). But it requires manual configuration, by default the standard approach is used. So the bottom-line is: Time will be forcefully stepped ahead by 1 second (which is an uncommon scenario on production server), and, it would not just be an ordinary step - it uses the operating systems normally untested negative leap second code path. I'm not sure what the consequences would be for various services, perhaps not much, but my worry is the interactions in a complex application can produce unexpected outcomes. Also, how many systems have implemented the negative leap second in the code path? And are they all tested?
- throwaway09223 6y ago
  Well, not really. Clock skew is tested, but the code paths in adjtimex() for handling a leap second are different. This is why the 2005 leap second crashed every Linux 2.4 kernel worldwide as I recall. Thankfully there's been a push toward leap smear (certainly on every system I run) which should obviate this type of problem entirely.
  - deleted 6y ago
    [deleted]
- tiagod 6y ago
  Does NTP do that, though? I was under the impression it made time slightly faster/slower until the point is met, if the difference is small (which it will always be on production servers, in principle)
  - throwaway09223 6y ago
    ntpd does indeed skip time, contrary to popular belief. I didn't believe it myself until one day I studied the documentation and code. Check out the "-x" flag in the manpage. I recommend dumping ntpd and using chrony instead, and also configuring chrony for a time smear rather than the leap second as required by the standard.
    - segfaultbuserr 6y ago
      > ntpd does indeed skip time The documentation also indicates it's rare for a 7x24 running server to suddenly step time (it only occurs when there's a 128 ms difference), unless you get serious network congestion that prevents sychroninization for an extended period, so a time step in a production server is uncommon. But a standard-compliant negative leap second requires time stepping, it's why I said there might be disruption.
      - deleted 6y ago
        [deleted]
      - throwaway09223 6y ago
        Yes, but the documentation is wrong in its assumption. There are many reasons time may skew >128ms beyond "network congestion." I've dealt with datacenters getting a bad ntp config push, for example, or a bad firewall ruleset which blocks NTP. Fixing the config may inadvertantly step time. There's a whole host of systems issues, sometimes involving VMs and sometimes bare metal which can cause this type of problem. The most simple of which is ntpd not running properly for a while, but this can also happen when systems get overloaded - extreme swapping and the like. ntpd's behavior of non-monotonic time adjustments was brought to my attention after a developer uncovered database corruption, caused by a database which (correctly) assumed that time should be monotonically adjusted while the database server was running. I believe it mis-ordered a transaction log or something along those lines. There are other, better ntpd implementations like chrony which can be configured to never step time. If time needs to be stepped, applications should be stopped and the machine should be removed from service first.
- LoSboccacc 6y ago
  a skip ahead is different than the officially being at 23:59:60, albeit quite many NTP nowaday use leap smearing to avoid that https://developers.google.com/time/smear https://developers.google.com/time/smear
throw0101a 6y ago
> But nobody has tested it in the wild. The FreeBSD (kernel) folks test their code for it: > I run the freebsd kernel and ntpd through "negative" leap second events usually a couple times a year as part of testing our timekeeping products at $work. I'm skeptical that we'll ever see a negative leap, but we have some customers who insist on having it demonstrated to them as working correctly. > […] * https://lists.freebsd.org/pipermail/freebsd-stable/2020-November/092837.html https://lists.freebsd.org/pipermail/freebsd-stable/2020-Nove... Of course then there's userland…
- segfaultbuserr 6y ago
  Great, at least we know the kernel is probably good (assuming Linux gets tested too and at least the latest version doesn't have bugs)... But I still worry about the interactions in a complex userspace application can produce unexpected outcomes. And I wonder whether Linux previously had bugs (too lazy to trace down the git history today), and if there was, how many are still in deployment...
eyelidlessness 6y ago
What I don’t understand is what the world gains by this level of global software time accuracy that couldn’t be localized to the very specialized cases that actually need it. Why can’t the world generally drift a second here and there over a few years and super time sensitive programs keep a tally of leap seconds to adjust? Sure it’s a pain in the ass for those applications, but isn’t “we don’t know for sure if this time adjustment is going to disrupt major world affairs” more of one?

jedberg 6y ago

It's constantly tested in the wild. Every time your computer does an NTP update and has to jump ahead one second or more, you've just tested it.

segfaultbuserr 6y ago
> Every time your computer does an NTP update and has to jump ahead one second PCs don't run anything crucial, usually you can set your clock to 2001 without visible problems, the same is not true for a server. A server uses ntpd, not a bruce-force ntpdate. Under ordinary conditions, ntpd adjusts the clock in small steps so that the timescale is effectively continuous and without discontinuities. But I think a negative leap second is quite violent.
- jedberg 6y ago
  I was talking about ntpd. It would do its update after the negative leap second and then think to itself "oh, I'm more than one second behind now, I better start fuzzing and catch up".
  - segfaultbuserr 6y ago
    It's rare for ntpd to step time forward, unless ntpd lost synchronization and the local clock has drifted beyond +/- 128 ms by default. On a 7x24 production server, it's very unlikely, time is always slewed. But a standard leap second implementation requires the addition or deletion of time via adjtimex()'s TIME_INS and TIME_DEL, it's not adjusted gradually. > "oh, I'm more than one second behind now, I better start fuzzing and catch up". No, ntpd would say, oh, leap second, now it's the time to tell kernel to delete a second! And if the kernel's negative leap second code path has a bug, kernel panics and everything crashes. ...Yes, you probably can force ntpd to ignore the leap second deletion and also force NTP to slew time only. This approach is similar to leap second smearing (which is considered a violation of standard but nevertheless an useful hack). But it requires manual configuration, by default the standard approach is used. So the bottom-line is: Time will be forcefully stepped ahead by 1 second (which is an uncommon scenario on production server), and, it would not just be an ordinary step - it uses the operating systems normally untested negative leap second code path. I'm not sure what the consequences would be for various services, perhaps not much, but my worry is the interactions in a complex application can produce unexpected outcomes. Also, how many systems have implemented the negative leap second in the code path? And are they all tested?
throwaway09223 6y ago
Well, not really. Clock skew is tested, but the code paths in adjtimex() for handling a leap second are different. This is why the 2005 leap second crashed every Linux 2.4 kernel worldwide as I recall. Thankfully there's been a push toward leap smear (certainly on every system I run) which should obviate this type of problem entirely.
- deleted 6y ago
  [deleted]
tiagod 6y ago
Does NTP do that, though? I was under the impression it made time slightly faster/slower until the point is met, if the difference is small (which it will always be on production servers, in principle)
- throwaway09223 6y ago
  ntpd does indeed skip time, contrary to popular belief. I didn't believe it myself until one day I studied the documentation and code. Check out the "-x" flag in the manpage. I recommend dumping ntpd and using chrony instead, and also configuring chrony for a time smear rather than the leap second as required by the standard.
  - segfaultbuserr 6y ago
    > ntpd does indeed skip time The documentation also indicates it's rare for a 7x24 running server to suddenly step time (it only occurs when there's a 128 ms difference), unless you get serious network congestion that prevents sychroninization for an extended period, so a time step in a production server is uncommon. But a standard-compliant negative leap second requires time stepping, it's why I said there might be disruption.
    - deleted 6y ago
      [deleted]
    - throwaway09223 6y ago
      Yes, but the documentation is wrong in its assumption. There are many reasons time may skew >128ms beyond "network congestion." I've dealt with datacenters getting a bad ntp config push, for example, or a bad firewall ruleset which blocks NTP. Fixing the config may inadvertantly step time. There's a whole host of systems issues, sometimes involving VMs and sometimes bare metal which can cause this type of problem. The most simple of which is ntpd not running properly for a while, but this can also happen when systems get overloaded - extreme swapping and the like. ntpd's behavior of non-monotonic time adjustments was brought to my attention after a developer uncovered database corruption, caused by a database which (correctly) assumed that time should be monotonically adjusted while the database server was running. I believe it mis-ordered a transaction log or something along those lines. There are other, better ntpd implementations like chrony which can be configured to never step time. If time needs to be stepped, applications should be stopped and the machine should be removed from service first.
LoSboccacc 6y ago
a skip ahead is different than the officially being at 23:59:60, albeit quite many NTP nowaday use leap smearing to avoid that https://developers.google.com/time/smear https://developers.google.com/time/smear

throw0101a 6y ago

> But nobody has tested it in the wild. The FreeBSD (kernel) folks test their code for it: > I run the freebsd kernel and ntpd through "negative" leap second events usually a couple times a year as part of testing our timekeeping products at $work. I'm skeptical that we'll ever see a negative leap, but we have some customers who insist on having it demonstrated to them as working correctly. > […] * https://lists.freebsd.org/pipermail/freebsd-stable/2020-November/092837.html https://lists.freebsd.org/pipermail/freebsd-stable/2020-Nove... Of course then there's userland…

segfaultbuserr 6y ago
Great, at least we know the kernel is probably good (assuming Linux gets tested too and at least the latest version doesn't have bugs)... But I still worry about the interactions in a complex userspace application can produce unexpected outcomes. And I wonder whether Linux previously had bugs (too lazy to trace down the git history today), and if there was, how many are still in deployment...

eyelidlessness 6y ago

What I don’t understand is what the world gains by this level of global software time accuracy that couldn’t be localized to the very specialized cases that actually need it. Why can’t the world generally drift a second here and there over a few years and super time sensitive programs keep a tally of leap seconds to adjust? Sure it’s a pain in the ass for those applications, but isn’t “we don’t know for sure if this time adjustment is going to disrupt major world affairs” more of one?