5 ms·

> NTP or kernel timekeeping code expects that it will be an appalling shitshow. I’m curious what systems would catastrophically fall apart if time went back on

by etxm 6y ago

> NTP or kernel timekeeping code expects that it will be an appalling shitshow. I’m curious what systems would catastrophically fall apart if time went back one second?

vimax 6y ago
About 10 years ago all Java processes everywhere deadlocked because of this.
- teraflop 6y ago
  Technically, that was a Linux kernel bug that Java happened to be heavily affected by: https://news.ycombinator.com/item?id=4188412 https://news.ycombinator.com/item?id=4188412
dudus 6y ago
Distributed alogrithms often depend on precise clocks to maintain consistency. See google spanner for instance. Updating the clock one second means taking a whole cluster offline untill you are sure the clocks are synchronized exactly before bringing them online again. It brings an outage and potential of headaches. If the clocks are not synchronized you break the consensus, might cause transactions to commit partially or cause inconsistent reads and writes.
- centimeter 6y ago
  Those algorithms should be using TAI, not UTC. Any distributed system that breaks in the presence of leap seconds is necessarily poorly designed. The fact that any non-UI code uses any time standard except TAI is a very unfortunate historical mistake that will hopefully be cleaned up over the next few years. Thankfully since 2013, Linux kernel has supported TAI.
  - lifthrasiir 6y ago
    > Thankfully since 2013, Linux kernel has supported TAI. Only with the correct ntpd configuration [1]. For that reason it is impossible to rely on that API in end-user libraries. [1] https://superuser.com/questions/1156693/is-there-a-way-of-getting-correct-clock-tai-on-linux https://superuser.com/questions/1156693/is-there-a-way-of-ge...
    - thayne 6y ago
      And from what I can tell it doesn't work with systemd-timesyncd, which many linux systems are using now.
  - brandmeyer 6y ago
    Pretty sure the Spanner case isn't using either one. Google's systems futz with the speed of time near the leap second to distribute it across a wide span about midnight, UTC. https://developers.google.com/time/smear https://developers.google.com/time/smear
  - beagle3 6y ago
    > The fact that any non-UI code uses any time standard except TAI is a very unfortunate historical mistake that will hopefully be cleaned up over the next few years. Unfortunately, almost everything is UI code in some way, so this won't be cleaned up. The problem with TAI is that you cannot convert a future time to UTC if it is farther than 6 months or so in the future (because leap seconds have not been decided), so unless it never needs any conversion to/from UTC, you can't use it. (are logs UI? is crontab UI? eventually everything is) Furthermore, you can't build an embedded no-need-to-update-every-6-month system with available common protocol - you would need to continuously feed it the list of leap-seconds, for which AFAIK there is no widely deployed protocol. But if you use UTC, there are many protocols (TIME, NTP, SNTP, PTP), at least one of which you'll use anyway because of clock drifts, that will let you keep working.
    - lifthrasiir 6y ago
      > The problem with TAI is that you cannot convert a future time to UTC if it is farther than 6 months or so in the future [...] Centimeter's argument is that such time should not be converted to TAI in advance, much like the local time for the future point (since the local time zone offset may change). All points on this also apply to the local time, e.g. it's already true that non-updatable embedded systems can't reliably do the local time calculation.
      - beagle3 6y ago
        They can’t do local time, but they can do UTC (and do it) without much investment if you run an NTP client. (Have 500 or so such units deployed, SNTP was giving too much trouble, switched to chrony at some point and those units require much less support)
rachelbythebay 6y ago
A negative leap second (the appalling shitshow from the article) would make us skip a second, though. Going back one second is what generally happens now when a Linux box does the "inserting leap second" thing. It goes from 23:59:59.999999 to 23:59:59.000000, then runs that whole second again. You get to 23:59:59.999999 again, and then you finally roll over to 00:00:00.000000. From the perspective of the typical time_t rendering of Unix time, there is no way to uniquely represent that 61st second. It just "disappears". Having lived through systems going backwards some 17 seconds due to a botched NTP-GPS appliance config, I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid. This didn't have to happen. They could have used monotonic from the get-go, and then they wouldn't have died when the clocks got yanked backwards 17 seconds to the proper time base.
- beagle3 6y ago
  > Having lived through systems going backwards some 17 seconds due to a botched NTP-GPS appliance config Well then, it really was botched. I was running an off-line hodge podge of 50 or so RH7.3 + Win2K systems back in 2002, where I had to manually fix the few second drift every couple of months, and the RH machines all slewed faster/slower to adapt, non ever went backward or jumped forward (Don't remember how the Win2K handled it; They weren't mission critical and I didn't care much).
- bogomipz 6y ago
  >'From the perspective of the typical time_t rendering of Unix time, there is no way to uniquely represent that 61st second. It just "disappears".' This is a great and really intuitive summary of the problem. Is there a certain class of problem related to disappearing second? Like are these more likely to be filesystem issues or things that rely on timestamps? Or are there second order problems as well? >"I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid. I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid." Sorry if this is a silly question but was that check simply that "time t1 is greater than time t0"? Also was the duration of that outage(17 seconds) or would this have been equally catastrophic at a single second?
spullara 6y ago
I was at Twitter when lots of machines seized. https://www.wired.com/2012/07/leap-second-glitch-explained/ https://www.wired.com/2012/07/leap-second-glitch-explained/
toast0 6y ago
Linux had two different kernel bugs around this, IIRC. MySQL got confused too (although that one just ate cpu until you restarted the daemon, it didn't crash, and it didn't prevent service)

by etxm 6y ago

> NTP or kernel timekeeping code expects that it will be an appalling shitshow. I’m curious what systems would catastrophically fall apart if time went back one second?

vimax 6y ago

About 10 years ago all Java processes everywhere deadlocked because of this.

teraflop 6y ago
Technically, that was a Linux kernel bug that Java happened to be heavily affected by: https://news.ycombinator.com/item?id=4188412 https://news.ycombinator.com/item?id=4188412

dudus 6y ago

Distributed alogrithms often depend on precise clocks to maintain consistency. See google spanner for instance. Updating the clock one second means taking a whole cluster offline untill you are sure the clocks are synchronized exactly before bringing them online again. It brings an outage and potential of headaches. If the clocks are not synchronized you break the consensus, might cause transactions to commit partially or cause inconsistent reads and writes.

centimeter 6y ago
Those algorithms should be using TAI, not UTC. Any distributed system that breaks in the presence of leap seconds is necessarily poorly designed. The fact that any non-UI code uses any time standard except TAI is a very unfortunate historical mistake that will hopefully be cleaned up over the next few years. Thankfully since 2013, Linux kernel has supported TAI.
- lifthrasiir 6y ago
  > Thankfully since 2013, Linux kernel has supported TAI. Only with the correct ntpd configuration [1]. For that reason it is impossible to rely on that API in end-user libraries. [1] https://superuser.com/questions/1156693/is-there-a-way-of-getting-correct-clock-tai-on-linux https://superuser.com/questions/1156693/is-there-a-way-of-ge...
  - thayne 6y ago
    And from what I can tell it doesn't work with systemd-timesyncd, which many linux systems are using now.
- brandmeyer 6y ago
  Pretty sure the Spanner case isn't using either one. Google's systems futz with the speed of time near the leap second to distribute it across a wide span about midnight, UTC. https://developers.google.com/time/smear https://developers.google.com/time/smear
- beagle3 6y ago
  > The fact that any non-UI code uses any time standard except TAI is a very unfortunate historical mistake that will hopefully be cleaned up over the next few years. Unfortunately, almost everything is UI code in some way, so this won't be cleaned up. The problem with TAI is that you cannot convert a future time to UTC if it is farther than 6 months or so in the future (because leap seconds have not been decided), so unless it never needs any conversion to/from UTC, you can't use it. (are logs UI? is crontab UI? eventually everything is) Furthermore, you can't build an embedded no-need-to-update-every-6-month system with available common protocol - you would need to continuously feed it the list of leap-seconds, for which AFAIK there is no widely deployed protocol. But if you use UTC, there are many protocols (TIME, NTP, SNTP, PTP), at least one of which you'll use anyway because of clock drifts, that will let you keep working.
  - lifthrasiir 6y ago
    > The problem with TAI is that you cannot convert a future time to UTC if it is farther than 6 months or so in the future [...] Centimeter's argument is that such time should not be converted to TAI in advance, much like the local time for the future point (since the local time zone offset may change). All points on this also apply to the local time, e.g. it's already true that non-updatable embedded systems can't reliably do the local time calculation.
    - beagle3 6y ago
      They can’t do local time, but they can do UTC (and do it) without much investment if you run an NTP client. (Have 500 or so such units deployed, SNTP was giving too much trouble, switched to chrony at some point and those units require much less support)

rachelbythebay 6y ago

A negative leap second (the appalling shitshow from the article) would make us skip a second, though. Going back one second is what generally happens now when a Linux box does the "inserting leap second" thing. It goes from 23:59:59.999999 to 23:59:59.000000, then runs that whole second again. You get to 23:59:59.999999 again, and then you finally roll over to 00:00:00.000000. From the perspective of the typical time_t rendering of Unix time, there is no way to uniquely represent that 61st second. It just "disappears". Having lived through systems going backwards some 17 seconds due to a botched NTP-GPS appliance config, I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid. This didn't have to happen. They could have used monotonic from the get-go, and then they wouldn't have died when the clocks got yanked backwards 17 seconds to the proper time base.

beagle3 6y ago
> Having lived through systems going backwards some 17 seconds due to a botched NTP-GPS appliance config Well then, it really was botched. I was running an off-line hodge podge of 50 or so RH7.3 + Win2K systems back in 2002, where I had to manually fix the few second drift every couple of months, and the RH machines all slewed faster/slower to adapt, non ever went backward or jumped forward (Don't remember how the Win2K handled it; They weren't mission critical and I didn't care much).
bogomipz 6y ago
>'From the perspective of the typical time_t rendering of Unix time, there is no way to uniquely represent that 61st second. It just "disappears".' This is a great and really intuitive summary of the problem. Is there a certain class of problem related to disappearing second? Like are these more likely to be filesystem issues or things that rely on timestamps? Or are there second order problems as well? >"I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid. I can tell you that what died at that particular site was all of the locking code that used wall time clocks instead of monotonic clocks. They all CHECKed and died when their preconditions were no longer valid." Sorry if this is a silly question but was that check simply that "time t1 is greater than time t0"? Also was the duration of that outage(17 seconds) or would this have been equally catastrophic at a single second?

spullara 6y ago

I was at Twitter when lots of machines seized. https://www.wired.com/2012/07/leap-second-glitch-explained/ https://www.wired.com/2012/07/leap-second-glitch-explained/

toast0 6y ago

Linux had two different kernel bugs around this, IIRC. MySQL got confused too (although that one just ate cpu until you restarted the daemon, it didn't crash, and it didn't prevent service)