7 ms·

> Instead of adding a single extra second to the end of the day, we'll run the clocks 0.0014% slower across the ten hours before and ten hours after the leap se

> Instead of adding a single extra second to the end of the day, we'll run the clocks 0.0014% slower across the ten hours before and ten hours after the leap second, and “smear” the extra second across these twenty hours. Holy leaping second, batman! Unilaterally being off by up to a half second from the rest of the world's clocks is a pretty aggressive step. I think I would have preferred to see a resolution made by an independent body on something this drastic.

subway 10y ago
They've been doing this since at least 2011.
- brandmeyer 10y ago
  That doesn't make it any less unilateral. IMO, it makes it rather worse to have been doing this for five years. The situation would be much better if they had been working to build a broader consensus over all that time. As near as I can tell, they don't even have consensus within Linux, let alone POSIX or the ITU.
  - detaro 10y ago
    Honest question: what's so bad about others' systems running on slightly off time? I get why people care about internal consistency, and why deviations should be quite small, but this?
    - brandmeyer 10y ago
      Slightly contrived example: Lets say that you were running a distributed database, and you had distributed instances across different cloud providers for increased reliability. if your database relies on high-resolution timestamps for distributed conflict resolution, then you're going to have a hard time. Another example: Suppose that a portion of an industrial monitoring system processes remote sensor data in a cloud datacenter with smeared time, while the sensor nodes keep strict UTC time. Your SCADA system had better not have any hard-baked assumptions like "messages cannot come from the future", or you're going to have a hard time, too. Lets say that a company's internal NTC servers include several sources for reliability and redundancy. Much like Google DNS, perhaps one of the sources is Google NTP, while another is derived from the NTP pool. How do you expect the NTP daemon to behave in this situation? It will certainly be able to observe a 500ms difference between its source timeservers.
      - detaro 10y ago
        first example: ok, yes, if they offer the DB as a service, that would be bad. If you run it on a VM it's IMHO your responsibility to make sure your time sensitive database nodes have shared time. Same for the second example, interesting point for SaaS scenario, although it seems like that could break through normal deviations already. EDIT: ok, the blog post actually mentions "local clocks in sync with VM instances running on Google Compute Engine", my bad. Not sure what to think about that. In comparison, Amazon recommends running NTP on your VMs and their Linux AMIs come with pool.ntp.org configured as default. </edit> Third: It's going to figure out some solution (if Google is only one source it's probably going to drop it as faulty), but you probably should not have added a time source that's officially documented to not strictly follow standards. It's not like Google offered a NTP service for years and now suddenly switched how it works. I guess I underestimate the amount of trust people put into random time sources: practice is probably messier than theory.
      - subway 10y ago
        Everywhere Google documents the service, they clearly state you should not mix their smearing NTP servers with non-smearing NTP servers.
        oasisbob 10y ago
        I'm not convinced it would do anything harmful, so long as you have enough NTP sources (which you should have anyway). From their FAQ: > We recommend that you do not mix smeared and non-smeared NTP servers. The results during a leap second may be unpredictable. I read that as a soft SHOULD NOT, not MUST NOT. Would be a fun exercise to try doing it intentionally with common NTP implementations and see what happens.
      - oasisbob 10y ago
        Both of those examples strike me as very contrived. I can't think of anyone who cares that much about timekeeping who isn't running their own internal NTP infrastructure. Google's Spanner requires accurate global time, so they deployed GPS and atomic clocks. Same for CDMA. There are some applications for high-resolution time (eg finance), so protocols like PTP exist. A smeared NTP source in an otherwise normal list of time sources doesn't seem like that big of a deal either - eventually the daemon is just going to mark it as a falseticker and life goes on.
    - toast0 10y ago
      During the last leap second, I had servers configured against google's semi-public servers and some other good sources of time. ntpd marked the google servers as a false ticker sometime during the distortion, and when it was done, was happy with it again. However, I have more non-google servers than google servers, and high minpoll times which tends to result in time checks between servers happening far apart in time, so even if I had multiple google servers, they wouldn't look very close together.
  - TheGrumpyBrit 10y ago
    In the absense of a published and agreed standard, every approach is unilateral. My company is taking a similar approach - disconnecting from external NTP servers on 31st December, stepping the change in gradually, and reconnecting when we're "right" again. Google have never been in the NTP business - there's no reason for them to have worked towards a concensus on this. But when a company their size makes their approach publicly available to all, it starts to pave the way for a consistent standard for everyone.
    - toast0 10y ago
      Google is in the NTP business. Chromebooks sync time from their servers, android devices can (but also from carrier provide time signals), and for hosts in their cloud services.
JoshTriplett 10y ago
Smearing leap seconds does make sense, but it's an odd step to take unilaterally, rather than coordinating with other NTP servers and with Linux timekeeping (which currently handles leap seconds via a 61-second minute instead).
- ac29 10y ago
  >Linux timekeeping (which currently handles leap seconds via a 61-second minute instead) Google doesn't think so: "No commonly used operating system is able to handle a minute with 61 seconds"
  - jmgao 10y ago
    Bits of pieces of the operating systems might handle leap seconds properly, but it's doubtful that every single component that uses time does the right thing. The last two leap seconds have revealed bugs in the kernel: https://lwn.net/Articles/504744/ https://lwn.net/Articles/504744/ for the one in 2015 and https://lwn.net/Articles/648313/ https://lwn.net/Articles/648313/ for the one in 2016, and I don't think it's unlikely that the one scheduled for December will reveal another.
  - JoshTriplett 10y ago
    Some things did go wrong on a few of the previous leap-second injections, and the Linux timekeeping maintainer had talked about changing the approach to handling them (which has already changed at least once in the past). I don't, however, think it makes sense to unilaterally change this, without (any obvious signs of) coordination with the timekeeping maintainers and the maintainers of major NTP servers.
    - fanf2 10y ago
      Things go wrong on every leap second, sometimes catastrophically. They go wrong on non-leap-seconds because of falsely advertised leap seconds. They go wrong 4 months before a leap second because a leap indicator got set and some software had an incorrect idea of when it was due. Never mind the theory, the practice is a clusterfuck.
KayEss 10y ago
It's far better than what POSIX clocks do. They'll just drop back a second and you'll get the same time twice.
- paulajohnson 10y ago
  That is a historical artifact. The original Unix developers decided to treat time as seconds since the start of 1970, implicitly assuming that every day has 86400 seconds. Back then UTC was in its infancy, most programmers had not even heard of leap seconds, and most computer clocks were set by the sysadmin looking at his watch. If we were starting from scratch we would have a date-time type with a day number field and a seconds-since-midnight field. However that would be a breaking change for every piece of software out there, so we are stuck with a time_t that cannot handle leap seconds.
  - leni536 10y ago
    Or we would use the NTP timestamp directly (or a lower precision version) as time_t, which (AFAIK) doesn't suffer from leap seconds. On can always convert time_t to truct tm, tm_sec is defined to be in the range of [0..60].
    - leni536 10y ago
      Looks like I was wrong: https://www.eecis.udel.edu/~mills/leap.html https://www.eecis.udel.edu/~mills/leap.html
magicalist 10y ago
You're going to have a bad time if you assume "the rest of the world" isn't doing their own, different adjustment https://developers.google.com/time/smear#othersmears https://developers.google.com/time/smear#othersmears > preferred to see a resolution made by an independent body Independent bodies have spent the last decade debating if leap seconds should even exist. Agreeing on how to treat them if we keep them is way down the priority list.
- orblivion 10y ago
  > You're going to have a bad time... Hilarious. Was this intentional?

subway 10y ago

They've been doing this since at least 2011.

brandmeyer 10y ago
That doesn't make it any less unilateral. IMO, it makes it rather worse to have been doing this for five years. The situation would be much better if they had been working to build a broader consensus over all that time. As near as I can tell, they don't even have consensus within Linux, let alone POSIX or the ITU.
- detaro 10y ago
  Honest question: what's so bad about others' systems running on slightly off time? I get why people care about internal consistency, and why deviations should be quite small, but this?
  - brandmeyer 10y ago
    Slightly contrived example: Lets say that you were running a distributed database, and you had distributed instances across different cloud providers for increased reliability. if your database relies on high-resolution timestamps for distributed conflict resolution, then you're going to have a hard time. Another example: Suppose that a portion of an industrial monitoring system processes remote sensor data in a cloud datacenter with smeared time, while the sensor nodes keep strict UTC time. Your SCADA system had better not have any hard-baked assumptions like "messages cannot come from the future", or you're going to have a hard time, too. Lets say that a company's internal NTC servers include several sources for reliability and redundancy. Much like Google DNS, perhaps one of the sources is Google NTP, while another is derived from the NTP pool. How do you expect the NTP daemon to behave in this situation? It will certainly be able to observe a 500ms difference between its source timeservers.
    - detaro 10y ago
      first example: ok, yes, if they offer the DB as a service, that would be bad. If you run it on a VM it's IMHO your responsibility to make sure your time sensitive database nodes have shared time. Same for the second example, interesting point for SaaS scenario, although it seems like that could break through normal deviations already. EDIT: ok, the blog post actually mentions "local clocks in sync with VM instances running on Google Compute Engine", my bad. Not sure what to think about that. In comparison, Amazon recommends running NTP on your VMs and their Linux AMIs come with pool.ntp.org configured as default. </edit> Third: It's going to figure out some solution (if Google is only one source it's probably going to drop it as faulty), but you probably should not have added a time source that's officially documented to not strictly follow standards. It's not like Google offered a NTP service for years and now suddenly switched how it works. I guess I underestimate the amount of trust people put into random time sources: practice is probably messier than theory.
    - subway 10y ago
      Everywhere Google documents the service, they clearly state you should not mix their smearing NTP servers with non-smearing NTP servers.
      - oasisbob 10y ago
        I'm not convinced it would do anything harmful, so long as you have enough NTP sources (which you should have anyway). From their FAQ: > We recommend that you do not mix smeared and non-smeared NTP servers. The results during a leap second may be unpredictable. I read that as a soft SHOULD NOT, not MUST NOT. Would be a fun exercise to try doing it intentionally with common NTP implementations and see what happens.
    - oasisbob 10y ago
      Both of those examples strike me as very contrived. I can't think of anyone who cares that much about timekeeping who isn't running their own internal NTP infrastructure. Google's Spanner requires accurate global time, so they deployed GPS and atomic clocks. Same for CDMA. There are some applications for high-resolution time (eg finance), so protocols like PTP exist. A smeared NTP source in an otherwise normal list of time sources doesn't seem like that big of a deal either - eventually the daemon is just going to mark it as a falseticker and life goes on.
  - toast0 10y ago
    During the last leap second, I had servers configured against google's semi-public servers and some other good sources of time. ntpd marked the google servers as a false ticker sometime during the distortion, and when it was done, was happy with it again. However, I have more non-google servers than google servers, and high minpoll times which tends to result in time checks between servers happening far apart in time, so even if I had multiple google servers, they wouldn't look very close together.
- TheGrumpyBrit 10y ago
  In the absense of a published and agreed standard, every approach is unilateral. My company is taking a similar approach - disconnecting from external NTP servers on 31st December, stepping the change in gradually, and reconnecting when we're "right" again. Google have never been in the NTP business - there's no reason for them to have worked towards a concensus on this. But when a company their size makes their approach publicly available to all, it starts to pave the way for a consistent standard for everyone.
  - toast0 10y ago
    Google is in the NTP business. Chromebooks sync time from their servers, android devices can (but also from carrier provide time signals), and for hosts in their cloud services.

JoshTriplett 10y ago

Smearing leap seconds does make sense, but it's an odd step to take unilaterally, rather than coordinating with other NTP servers and with Linux timekeeping (which currently handles leap seconds via a 61-second minute instead).

ac29 10y ago
>Linux timekeeping (which currently handles leap seconds via a 61-second minute instead) Google doesn't think so: "No commonly used operating system is able to handle a minute with 61 seconds"
- jmgao 10y ago
  Bits of pieces of the operating systems might handle leap seconds properly, but it's doubtful that every single component that uses time does the right thing. The last two leap seconds have revealed bugs in the kernel: https://lwn.net/Articles/504744/ https://lwn.net/Articles/504744/ for the one in 2015 and https://lwn.net/Articles/648313/ https://lwn.net/Articles/648313/ for the one in 2016, and I don't think it's unlikely that the one scheduled for December will reveal another.
- JoshTriplett 10y ago
  Some things did go wrong on a few of the previous leap-second injections, and the Linux timekeeping maintainer had talked about changing the approach to handling them (which has already changed at least once in the past). I don't, however, think it makes sense to unilaterally change this, without (any obvious signs of) coordination with the timekeeping maintainers and the maintainers of major NTP servers.
  - fanf2 10y ago
    Things go wrong on every leap second, sometimes catastrophically. They go wrong on non-leap-seconds because of falsely advertised leap seconds. They go wrong 4 months before a leap second because a leap indicator got set and some software had an incorrect idea of when it was due. Never mind the theory, the practice is a clusterfuck.

KayEss 10y ago

It's far better than what POSIX clocks do. They'll just drop back a second and you'll get the same time twice.

paulajohnson 10y ago
That is a historical artifact. The original Unix developers decided to treat time as seconds since the start of 1970, implicitly assuming that every day has 86400 seconds. Back then UTC was in its infancy, most programmers had not even heard of leap seconds, and most computer clocks were set by the sysadmin looking at his watch. If we were starting from scratch we would have a date-time type with a day number field and a seconds-since-midnight field. However that would be a breaking change for every piece of software out there, so we are stuck with a time_t that cannot handle leap seconds.
- leni536 10y ago
  Or we would use the NTP timestamp directly (or a lower precision version) as time_t, which (AFAIK) doesn't suffer from leap seconds. On can always convert time_t to truct tm, tm_sec is defined to be in the range of [0..60].
  - leni536 10y ago
    Looks like I was wrong: https://www.eecis.udel.edu/~mills/leap.html https://www.eecis.udel.edu/~mills/leap.html

magicalist 10y ago

You're going to have a bad time if you assume "the rest of the world" isn't doing their own, different adjustment https://developers.google.com/time/smear#othersmears https://developers.google.com/time/smear#othersmears > preferred to see a resolution made by an independent body Independent bodies have spent the last decade debating if leap seconds should even exist. Agreeing on how to treat them if we keep them is way down the priority list.

orblivion 10y ago
> You're going to have a bad time... Hilarious. Was this intentional?