3 ms·

Fake leap seconds being announced by NTP servers, still causing linux crashes

51 points by 0x0 14y ago

NelsonMinar 14y ago
No need to panic; this event happend a few days ago, August 1. And no one's certain, but it seems just as likely this is a bug in NTP as it is a deliberate attempt to crash unpatched Linux servers. The way ntpd manages the leap second flag is pretty complex and error prone. Link to thread view of NTP list discussion: http://lists.ntp.org/pipermail/questions/2012-August/thread.html#33611 http://lists.ntp.org/pipermail/questions/2012-August/thread....
- mrud 14y ago
  From http://lists.ntp.org/pipermail/questions/2012-August/033626.html http://lists.ntp.org/pipermail/questions/2012-August/033626.... There are over 10 stratum one's that advertise LI=1 as of Wed Aug 1 14:18:51 UTC 2012. Unless this changes another false leap second could occur on August 31, 2012 So yes this was happening on July 31st but may happen as well on August 31.
- 0x0 14y ago
  There is a need to worry, because AFAIK at least Debian hasn't provided patches yet, so come September servers might still be crashing. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679882 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679882
- xfs 14y ago
  The fix needs backporting large patchset, yet the next leap second isn't announced. So no need to rush... just wait for a while and do QA. Typical mentality.
bjork 14y ago
Exactly how incompetent are linux coders? After all, the "problem" is extremely trivial.
- justinhj 14y ago
  Feel free to patch it, I'm sure the world of linux coders would be happy to have someone who finds this sort of matter trivial in their midst.
  - bjork 14y ago
    Well, I'm not a linux coder, but I'll give it a try and report back if I'm successful.
    - sp332 14y ago
      I think it's already been patched. https://lkml.org/lkml/2012/7/17/546 https://lkml.org/lkml/2012/7/17/546 (navigation on the left of the page)
      - 0x0 14y ago
        Looks like a long, 11-part patch set. Would be interesting to hear back from 'bjork' after his try to see if it is still trivial ;)
        sp332 14y ago
        https://lkml.org/lkml/2012/7/1/25 https://lkml.org/lkml/2012/7/1/25 Sounds like a race condition which is pretty awkward to avoid.
- zyb09 14y ago
  DateTime code bugs happen all the time, there are so much things to consider and it's hard to test, because the problem conditions might not occur for years to come. Remember this? http://news.cnet.com/new-years-hangover-for-zune-users/ http://news.cnet.com/new-years-hangover-for-zune-users/ (also I too am guilty of producing a similar bug in a software a few years ago)
gus_massa 14y ago
An idea: Setup a fake NTP server that inserts leap seconds randomly (for example, once a week) (forward and backward). It could be use with test servers to test if they are working correctly, before a real leap second crash the production server.
- hatcravat 14y ago
  Leap seconds only occur at the end of a month, so an NTP client that accepted such leapseconds would be non-compliant.

51 points by 0x0 14y ago

No need to panic; this event happend a few days ago, August 1. And no one's certain, but it seems just as likely this is a bug in NTP as it is a deliberate attempt to crash unpatched Linux servers. The way ntpd manages the leap second flag is pretty complex and error prone. Link to thread view of NTP list discussion: http://lists.ntp.org/pipermail/questions/2012-August/thread.html#33611 http://lists.ntp.org/pipermail/questions/2012-August/thread....

mrud 14y ago
From http://lists.ntp.org/pipermail/questions/2012-August/033626.html http://lists.ntp.org/pipermail/questions/2012-August/033626.... There are over 10 stratum one's that advertise LI=1 as of Wed Aug 1 14:18:51 UTC 2012. Unless this changes another false leap second could occur on August 31, 2012 So yes this was happening on July 31st but may happen as well on August 31.
0x0 14y ago
There is a need to worry, because AFAIK at least Debian hasn't provided patches yet, so come September servers might still be crashing. http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679882 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=679882
xfs 14y ago
The fix needs backporting large patchset, yet the next leap second isn't announced. So no need to rush... just wait for a while and do QA. Typical mentality.

bjork 14y ago

Exactly how incompetent are linux coders? After all, the "problem" is extremely trivial.

justinhj 14y ago
Feel free to patch it, I'm sure the world of linux coders would be happy to have someone who finds this sort of matter trivial in their midst.
- bjork 14y ago
  Well, I'm not a linux coder, but I'll give it a try and report back if I'm successful.
  - sp332 14y ago
    I think it's already been patched. https://lkml.org/lkml/2012/7/17/546 https://lkml.org/lkml/2012/7/17/546 (navigation on the left of the page)
    - 0x0 14y ago
      Looks like a long, 11-part patch set. Would be interesting to hear back from 'bjork' after his try to see if it is still trivial ;)
      - sp332 14y ago
        https://lkml.org/lkml/2012/7/1/25 https://lkml.org/lkml/2012/7/1/25 Sounds like a race condition which is pretty awkward to avoid.
zyb09 14y ago
DateTime code bugs happen all the time, there are so much things to consider and it's hard to test, because the problem conditions might not occur for years to come. Remember this? http://news.cnet.com/new-years-hangover-for-zune-users/ http://news.cnet.com/new-years-hangover-for-zune-users/ (also I too am guilty of producing a similar bug in a software a few years ago)

gus_massa 14y ago

An idea: Setup a fake NTP server that inserts leap seconds randomly (for example, once a week) (forward and backward). It could be use with test servers to test if they are working correctly, before a real leap second crash the production server.

hatcravat 14y ago
Leap seconds only occur at the end of a month, so an NTP client that accepted such leapseconds would be non-compliant.