4 ms·

The leap second has a history of breaking computers. Pretty much every Linux server running Java broke in 2012, for example. 2009 saw a bunch of Linux kernels c

The leap second has a history of breaking computers. Pretty much every Linux server running Java broke in 2012, for example. 2009 saw a bunch of Linux kernels crashing in logging code. 2016 was mostly better except for Cloudflare: their systems had an assumption time never runs backwards. https://blog.cloudflare.com/how-and-why-the-leap-second-affected-cloudflare-dns/ https://blog.cloudflare.com/how-and-why-the-leap-second-affe...

RcouF1uZ4gsC 8y ago
This is why some Google and AWS smear the leap second. https://developers.google.com/time/smear https://developers.google.com/time/smear https://aws.amazon.com/blogs/aws/look-before-you-leap-the-coming-leap-second-and-aws/ https://aws.amazon.com/blogs/aws/look-before-you-leap-the-co...
- chopin 8y ago
  Imho that's no better. A leap second "smears" the day lengths. Smearing the leap second means introducing changes at some scale, eg. more or less micro- or nanoseconds. Software needs ro be able to cope with this.
  - rkagerer 8y ago
    I had the same thought but... The smear technique results in seconds that are 11.6μs longer than nominal, and Google makes the case this is within the "manufacturing and thermal errors of most machines' quartz oscillators". So at face value, I'd say software already copes with discrepancies of that order. However I would infer the maximum delta actually seen by an NTP client is larger and depends on the client's polling interval. e.g. If a machine runs a normal clock locally and synchronizes every 20 minutes (which is probably on the high side), it would see about a 14ms jump on each sync during the 24 hour smear window. That's assuming the NTP client doesn't already use a smear algorithm locally to correct for drift (Windows doesn't, but I think Red Hat Linux might? Per https://access.redhat.com/solutions/39194 https://access.redhat.com/solutions/39194 "NTP polling does not directly synchronize the local system clock to the server clock; rather, a complex algorithm calculates an adjustment value for each tick of the local system clock") Maybe someone more well-versed can elaborate further.
ggm 8y ago
Not wanting to be a complete BSD bigot, I can't actually remember if this broke in the {Free,Net,Open}BSD world as badly. The 2015 docs for FreeBSD don't help (oh, for a time machine) but go to "you can test what we do" which is good. https://www.freebsd.org/doc/en_US.ISO8859-1/articles/leap-seconds/article.html https://www.freebsd.org/doc/en_US.ISO8859-1/articles/leap-se...
- ajross 8y ago
  Not to jump on the bigotry, but two of the three bugs mentioned in that comment were in proprietary software (Java and RRDNS) that merely runs on Linux. The hrtimer issue in 2012 was a real kernel bug though.
  - repolfx 8y ago
    Java was open sourced in 2006.
  - NelsonMinar 8y ago
    The 2012 bug was a bad interaction between Java and thread locking in the Linux kernel. It affected MySQL as well. The 2009 bug was in the Linux kernel, in the logging code. The 2016 bug I mentioned was in Cloudflare's proprietary code and only had a big impact because so many websites use Cloudflare. There are zillions of other smaller leap second related bugs, these are only three of the most visible. (btw, the word "bigotry" has a real meaning and this isn't it.)
- ars 8y ago
  BSD is actually worse, it doesn't handle leap seconds at all, it simply ignores their existence and assumes the clock was simply inaccurate. That's fine as far as it goes, but make sure never to use a BSD machine as an NTP server or you will cause a lot of confusion for any servers downstream.
  - swills 8y ago
    I don't know about OpenBSD or NetBSD but I've witnessed FreeBSD handling leap second correctly. I also witnessed OpenNTPD ignoring it and that's documented, so I suspect OpenBSD doesn't handle leap seconds at all.
Animats 8y ago
There's now "Google time".[1] This handles by leap seconds by making each second slightly longer, starting 12 hours before the leap second and ending 12 hours after it. [1] https://developers.google.com/time/smear https://developers.google.com/time/smear
- kbenson 8y ago
  While I wouldn't want to be the one responsible for designing, testing, implementing and converting to using said system, it does seem like the easiest way to handle the problem for the majority of cases in this day and age, where measuring 1/86,400th of a second is trivial. If only we were deciding how to implement leap seconds for the first time right now...
- bloak 8y ago
  And there's also UTC-SLS. But a more important distinction than whether you smear over +/-12 hours or +/-1000 seconds is whether you broadcast a bogus time signal to all your hapless servers, which is what Google does, I think, or whether you fix your kernel and C library so that the system knows the real time but reports UTC-SLS to those programs which use the old APIs and are therefore presumed to be incapable of understanding the real time. For example, clock_gettime(CLOCK_REALTIME, &res) might provide UTC-SLS (or whatever) for naive programs while the same syscall with CLOCK_TAI or CLOCK_UTC might give you the actual real time correctly.
  - lathiat 8y ago
    Old APIs won't be the only problems, it's easy to introduce bugs like this new API or not (it's much easier to say use monotonic timers these days, but it's still easy for devs not to use th=em)

RcouF1uZ4gsC 8y ago

This is why some Google and AWS smear the leap second. https://developers.google.com/time/smear https://developers.google.com/time/smear https://aws.amazon.com/blogs/aws/look-before-you-leap-the-coming-leap-second-and-aws/ https://aws.amazon.com/blogs/aws/look-before-you-leap-the-co...

chopin 8y ago
Imho that's no better. A leap second "smears" the day lengths. Smearing the leap second means introducing changes at some scale, eg. more or less micro- or nanoseconds. Software needs ro be able to cope with this.
- rkagerer 8y ago
  I had the same thought but... The smear technique results in seconds that are 11.6μs longer than nominal, and Google makes the case this is within the "manufacturing and thermal errors of most machines' quartz oscillators". So at face value, I'd say software already copes with discrepancies of that order. However I would infer the maximum delta actually seen by an NTP client is larger and depends on the client's polling interval. e.g. If a machine runs a normal clock locally and synchronizes every 20 minutes (which is probably on the high side), it would see about a 14ms jump on each sync during the 24 hour smear window. That's assuming the NTP client doesn't already use a smear algorithm locally to correct for drift (Windows doesn't, but I think Red Hat Linux might? Per https://access.redhat.com/solutions/39194 https://access.redhat.com/solutions/39194 "NTP polling does not directly synchronize the local system clock to the server clock; rather, a complex algorithm calculates an adjustment value for each tick of the local system clock") Maybe someone more well-versed can elaborate further.

ggm 8y ago

Not wanting to be a complete BSD bigot, I can't actually remember if this broke in the {Free,Net,Open}BSD world as badly. The 2015 docs for FreeBSD don't help (oh, for a time machine) but go to "you can test what we do" which is good. https://www.freebsd.org/doc/en_US.ISO8859-1/articles/leap-seconds/article.html https://www.freebsd.org/doc/en_US.ISO8859-1/articles/leap-se...

ajross 8y ago
Not to jump on the bigotry, but two of the three bugs mentioned in that comment were in proprietary software (Java and RRDNS) that merely runs on Linux. The hrtimer issue in 2012 was a real kernel bug though.
- repolfx 8y ago
  Java was open sourced in 2006.
- NelsonMinar 8y ago
  The 2012 bug was a bad interaction between Java and thread locking in the Linux kernel. It affected MySQL as well. The 2009 bug was in the Linux kernel, in the logging code. The 2016 bug I mentioned was in Cloudflare's proprietary code and only had a big impact because so many websites use Cloudflare. There are zillions of other smaller leap second related bugs, these are only three of the most visible. (btw, the word "bigotry" has a real meaning and this isn't it.)
ars 8y ago
BSD is actually worse, it doesn't handle leap seconds at all, it simply ignores their existence and assumes the clock was simply inaccurate. That's fine as far as it goes, but make sure never to use a BSD machine as an NTP server or you will cause a lot of confusion for any servers downstream.
- swills 8y ago
  I don't know about OpenBSD or NetBSD but I've witnessed FreeBSD handling leap second correctly. I also witnessed OpenNTPD ignoring it and that's documented, so I suspect OpenBSD doesn't handle leap seconds at all.

Animats 8y ago

There's now "Google time".[1] This handles by leap seconds by making each second slightly longer, starting 12 hours before the leap second and ending 12 hours after it. [1] https://developers.google.com/time/smear https://developers.google.com/time/smear

kbenson 8y ago
While I wouldn't want to be the one responsible for designing, testing, implementing and converting to using said system, it does seem like the easiest way to handle the problem for the majority of cases in this day and age, where measuring 1/86,400th of a second is trivial. If only we were deciding how to implement leap seconds for the first time right now...
bloak 8y ago
And there's also UTC-SLS. But a more important distinction than whether you smear over +/-12 hours or +/-1000 seconds is whether you broadcast a bogus time signal to all your hapless servers, which is what Google does, I think, or whether you fix your kernel and C library so that the system knows the real time but reports UTC-SLS to those programs which use the old APIs and are therefore presumed to be incapable of understanding the real time. For example, clock_gettime(CLOCK_REALTIME, &res) might provide UTC-SLS (or whatever) for naive programs while the same syscall with CLOCK_TAI or CLOCK_UTC might give you the actual real time correctly.
- lathiat 8y ago
  Old APIs won't be the only problems, it's easy to introduce bugs like this new API or not (it's much easier to say use monotonic timers these days, but it's still easy for devs not to use th=em)