8 ms·
Look Before You Leap – The Coming Leap Second and AWS
- ColinDabritz 11y agoIt's impressive the chaos that one second can cause. I think in terms of hard problems, to naming and cache invalidation, I would add time or human notions thereof. Or perhaps time is a case of the first two intersecting...
- erichurkman 11y agoDates, times, and time zones, +04:00 tough problems at the intersection of computer science and humans.
- WalterGR 11y agoPostal addresses, naming humans, encoding written language... Most problems at the intersection of technology and humans are tough. :)
- ComputerGuru 11y agoWow, they've certainly put a lot of thought into that! I wonder if they're actually "slowing down time" or just not implementing the leap second and just injecting the 1/..... second into all user-visible or exported fields?
- javert 11y agoI haven't seen a patent application for a time machine, so we can rule out the possibility that they're actually slowing down time. However, I do wonder how they implement this. Do you change timekeeping in the kernel so that a fraction of extra time must pass for each second to be "counted"?
- kccqzy 11y agoIt's probably just a patched NTP server that lies about current time.
- archimedespi 11y agoYou are exactly correct. The aforementioned thread[0] on Serverfault mentioned timesmearing with `ntpd -x`, as per [1]. It's a fairly well-known technique for avoiding some weird kernel-related bugs on leap seconds. [0]: http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second http://serverfault.com/questions/403732/anyone-else-experien... [1]: https://web.archive.org/web/20140221084401/http://my.opera.com/marcomarongiu/blog/2012/06/01/an-humble-attempt-to-work-around-the-leap-second https://web.archive.org/web/20140221084401/http://my.opera.c...
- graiz 11y agoInteresting. I remember reading how Google was doing something similar in their implementation of the leap second back in 2011: http://googleblog.blogspot.com/2011/09/time-technology-and-leaping-seconds.html http://googleblog.blogspot.com/2011/09/time-technology-and-l...
- stygiansonic 11y agoThe last time a leap second was added, it caused issues with a number of servers running Linux due to a livelock. [0] This had an effect on EC2 [1] and apparently other pieces of software [2] as well. Nice to see Amazon being proactive about the forthcoming change. Time is just a prime example of one of those things that seems superficially simple, but turns out to be deviously complicated. (If you ever want to bore non-technical types to death at a party, just start talking about the history of time. Bonus points for mentioning the proleptic Gregorian calendar in context.) 0. http://serverfault.com/questions/403732/anyone-else-experiencing-high-rates-of-linux-server-crashes-during-a-leap-second http://serverfault.com/questions/403732/anyone-else-experien... 1. http://www.techspot.com/news/49229-leap-second-bug-amazon-ec2-outage-brought-down-major-websites.html http://www.techspot.com/news/49229-leap-second-bug-amazon-ec... 2. http://www.wired.com/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/ http://www.wired.com/2012/07/leap-second-bug-wreaks-havoc-wi...
- brongondwana 11y ago0. That was an exciting day! I still remember rebooting servers on my phone while my kids tried on new bathers. http://blog.fastmail.com/2012/07/03/a-story-of-leaping-seconds/ http://blog.fastmail.com/2012/07/03/a-story-of-leaping-secon...
- cperciva 11y agoThis had an effect on EC2 I don't think the leap second bug had any effect on EC2 itself; the only reason EC2 was ever blamed is that many of the sites which had issues happened to be hosted on EC2.
- stygiansonic 11y agoCorrect, my bad. Thanks for catching the mistake.
- rolux 11y ago> just start talking about the history of time And lets not forget: 3. http://www.quirksmode.org/blog/archives/2009/04/making_time_saf.html http://www.quirksmode.org/blog/archives/2009/04/making_time_...
- pavel_lishin 11y agoI'm not sure if the addition of a leap second is what prompted qntm to write about this, but his twitter feed is worth checking out: https://twitter.com/qntm https://twitter.com/qntm
- wging 11y agoSee also his website/blog, where he often writes about time-travel-related ideas: http://qntm.org/ http://qntm.org/
- ggreer 11y agoI've always wondered why system clocks are set to UTC instead of TAI[1]. To me, it makes more sense for OSes to ship UTC as a time zone. Leap seconds would then be tzinfo updates, just like when countries change their daylight saving time. System clocks still wouldn't be guaranteed to be monotonically increasing, but at least there wouldn't be minutes with 61 seconds. 1. http://en.wikipedia.org/wiki/International_Atomic_Time http://en.wikipedia.org/wiki/International_Atomic_Time
- deathanatos 11y agoThis is certainly an interesting idea, but isn't TAI/UTC somewhat orthogonal to TZs? I'm in America/Los_Angeles, for example, and if I want my time in TAI, my local time would be ~35 seconds different from my local time in UTC, would it not?
- ggreer 11y agoLocal time zones are offset from UTC, not TAI. If it's 19:45:00 in LA, UTC is 02:45:00 and TAI is 02:45:35. There's no such thing as a "TAI version" of your local time.
- techdragon 11y agoThe solution is simple math, for computers to base their time on TAI, the time zone conversion changes from "UTC + local timezone offset", to "TAI + UTC offset + local timezone offset" and we reap the rewards of drastically simpler software at the core of our systems. TAI is defined like UNIX time, as a notation of the progression of proper time. It is the primary reference by which we build all other times, UTC is a humanist overlay on TAI to maintain norms, since we need an approximate terrestrial solar time for sanity purposes. If the math changes to TAI as the "base storage representation" for time stamps and reference time internally, then the math becomes immediately sane, since TAI can be relied on as a direct sequence of mathematically related linear time without lookup tables or other crap. Move the crap "up the stack" to where it doesn't cause issues like these we see every time things need a leap second.
- javert 11y agoWhat an elegant solution. This is really cool.
- maerF0x0 11y agoI'd like to hear from other HN readers if their applications would have major issues if they did not deal with leap seconds. Curious the domains in which this is very important.
- oh_sigh 11y agoYou could probably find a number of post mortems by googling around for the last leap second, which was June 30 2012 IIRC.
- TheLoneWolfling 11y agoWell, there was this: https://lkml.org/lkml/2012/7/1/203 https://lkml.org/lkml/2012/7/1/203 It affected pretty much everything running (a recent version of) Linux.
- marcosdumay 11y agoThat's a bug on code that handles leap second. It's not a bug triggered by losing accuracy by ignoring leap seconds. I'm also very curious about what kind of applications will fail if the kernel simply ignored leap seconds, and let the clock get a second out of sync. A computer does not last for long enough for leap seconds to add to anything. I'm unable to come out with any hypothetical use case where that small difference is relevant, but specialized hardware and software to deal with the issue are not necessary.
- onion2k 11y agohttps://github.com/search?q=86400&type=Code&utf8=✓ https://github.com/search?q=86400&type=Code&utf8=✓ A quick search on GitHub finds more than two million examples of people assuming 1 day is 86400 seconds. Every single one of those is a bug. Not a critical one (in most cases), but still an example of code that could fail unexpectedly. In the most basic case, code to fetch a date n days ahead that naively assumes fixed length days will return a date n-1 days ahead instead if it runs at exactly midnight eg date("Y-m-d", mktime()+($days*86400)) will return a date $days-1 in the future if it runs on leap second day. An edge case certainly, but if you're adding millions of records a day something you ought to consider.
- wslh 11y agoWhat a coincidence, today I was talking with the people at Vornex who are in the niche area of simulated time for testing. It seems like a good opportunity to test this "leap second". [1] http://www.vornexinc.com/ http://www.vornexinc.com/
- jpatokal 11y agoMildly disappointed that AWS isn't giving Google credit for coming up with the idea of "time smearing", which they're using here: http://googleblog.blogspot.com.au/2011/09/time-technology-and-leaping-seconds.html http://googleblog.blogspot.com.au/2011/09/time-technology-an...
- jasonmp85 11y agoAgreed that they definitely should have given some credit. I was unfamiliar with the concept (and didn't even know the term). The AWS post raised more questions than it answered, but that Google one is excellent and coins a great term.
- hillsarealiv3 11y agoThe idea of spreading a leap second over a longer period is not unique to Google e.g. www.quadibloc.com/science/cal06.htm The idea itself is actually rather obvious.
- jpatokal 11y agoMost good ideas are obvious in hindsight. ;) But AFAIK, Google was the first to do this live and in production by hacking NTP, which is (apparently?) the same as what Amazon is doing.
- foton1981 11y agoWhat if I disable ntpd a minute before leap second is injected and restart it a minute later?
- cnvogel 11y agoMy guess is, the leap second will be inserted, just as scheduled. And you'll get the famous message in your kernel log (dmesg): http://lxr.free-electrons.com/source/kernel/time/ntp.c?v=2.6.34#L199 http://lxr.free-electrons.com/source/kernel/time/ntp.c?v=2.6... Clock: inserting leap second 23:59:60 UTC If ntpd gets notified of an impeding leap second via its peers (or the connected radio clock, GPS receiver, ...) it will set (struct timex*)->status |= STA_INS via the timex syscall (which it also uses to steer/speed-up/slow-down the clock). http://man7.org/linux/man-pages/man2/adjtimex.2.html http://man7.org/linux/man-pages/man2/adjtimex.2.html The stock linux kernel works such that if ntpd will have set the STA_INS flag via adjtimex some time before, the kernel will do the leap-second insertion at the end of the UTC day. If you disable ntpd and it doesn't reset this flag (which I doubt it does, but you'd have to check), the kernel will insert the leap second on its own, even if ntpd is not running. If you disable ntpd, and either ntpd on termination (which I doubt), or you via the adjtimex syscall, clear the STA_INS flag, then the kernel will not insert the leap second. After UTC midnight, the clock will then be one second off, and a restart of ntpd will slowly steer the clock back to correct time. For playing with all of this, there's a adjtimex tool which can display and even change the timex values: ➜ sbin ./adjtimex -pV | sed 's/^/ /' mode: 0 offset: 0 frequency: -344608 maxerror: 16000000 esterror: 16000000 status: 64 time_constant: 2 precision: 1 tolerance: 32768000 tick: 10000 raw time: 1432018768s 308111us = 1432018768.308111 return value = 5
- foton1981 11y agoThanks! that's good info. It helped me to find a good article from someone to devoted couple days on testing this method: http://syslog.me/2012/06/01/an-humble-attempt-to-work-around-the-leap-second/comment-page-1/ http://syslog.me/2012/06/01/an-humble-attempt-to-work-around...
- deleted 11y ago[deleted]
- acqq 11y agoIf I understood the text correctly, AWS adjustments are "linear" and not curved like in the Google adjustment: lie(t) = (1.0 - cos(pi * t / w)) / 2.0 described here: http://googleblog.blogspot.com.au/2011/09/time-technology-and-leaping-seconds.html http://googleblog.blogspot.com.au/2011/09/time-technology-an... I think I'd prefer the curved one. I don't see the advantages of the linear one except the less human capability needed to implement and test it.
- bjackman 11y agoWhat are the advantages of the curved one? I've never worked on an application that would care about this kind of thing, so I'm struggling to imagine the implications.
- tarpherder 11y agoI'm guessing some applications may be sensitive to a sudden change in second duration and they probably argued that a gradual introduction of change would trip up the least amount of programs.
- TheLoneWolfling 11y agoSuppose you have something with a watchdog that times something every time it runs to check that nothing suddenly gets slower / faster. A backup or something. Let's suppose that the amount of time it takes to run is relatively small. Now suppose between two runs the linear decrease starts. All of a sudden the second run appears to be slower than it is by a constant factor. With a curved change, this effect still happens, but it's less for smaller time intervals.
- acqq 11y agoThe shortest explanation would be that the curve can be introduced to lower the average difference between the official time and the adjusted time. bjackman "never worked on an application that would care about this kind of thing," fine, but for those that do care, the curve is obviously better.
- acqq 11y agoAlso interesting: http://www.datastax.com/dev/blog/preparing-for-the-leap-second http://www.datastax.com/dev/blog/preparing-for-the-leap-seco...
- natch 11y agoI'm really doubting myself over this question, because who am I to think I might have found a mistake in Amazon's chart, but aren't these two comments in the table reversed? 1. "Each second is 1/86400 longer and AWS clocks fall behind UTC. The gap gradually increases to up to 1/2 second." If the seconds are longer, wouldn't the AWS clocks be creeping ahead, not falling behind? Bear in mind the leap second hasn't been added to UTC yet at this point in the table. 2. "AWS clocks gain 1/2 second ahead of UTC." They do? But an entire leap second was just added to UTC. Aren't the AWS clocks 1/2 second BEHIND at this point? Note the time of the AWS smearing of this 1 second straddles the UTC injection with 12 hours on each side. And then: "AWS clocks keep falling behind and the gap with UTC shrinks gradually." Shouldn't it be "AWS clocks keep catching up and the gap with UTC shrinks gradually"? Not sure how I can be reading this so backwards, but if it's me who is wrong here, I'd love to hear why. In any case it shows how time is easy to get wrong (whether it's me who is wrong, or hah, doubtful, AWS). What am I missing here? Edit: duh yeah I get it now. Amazon is right, of course. Obviously I'm a n00b when it comes to dealing with leap seconds. I was thinking the leap second sets time ahead, but it doesn't; it effectively does the opposite. Leaving this message in place as an example of how thinking about time and calendar-related programming is easy to mess up.
- abakker 11y agoThink of it this way. Over the next time interval, the true UTC click ticks 10 times. Then the time becomes time+10. If the AWS clock has longer seconds and only ticks 9 times, then the time is only time+9 on their clocks. Therefore, the time at AWS is falling behind UTC, since UTC is incrementing faster.
- stygiansonic 11y agoI really do find thinking about time to be fiendish as well. Here's some examples I thought of that may help clarify: 1. "Each second is 1/86400 longer and AWS clocks fall behind UTC. The gap gradually increases to up to 1/2 second." - AWK clocks do indeed fall-behind when their "seconds" tick longer. Because their seconds tick longer, over a fixed period of time, AWS will count less ticks than UTC. (Think of it this way: A mile is longer than a kilometer. After you have traveled 800 kilometers, you've only traveled ~500 miles.) 2. "AWS clocks gain 1/2 second ahead of UTC." - Before the addition of the leap second, AWS clocks are behind by 1/2 second as per (1). The addition of a leap second to UTC is another second that the UTC clock must tick - the AWS clocks don't have to tick this amount - so the AWS clocks are now ahead by 1/2 second. Basically, this is what AWS is doing, using our distance analogy again: 1. Normally, we have to cover 1000 km every day. This includes a Civil group of travelers and an AWS group of travelers. 2. Today, we decide we're going to cover 1001 km. Everyone in Civil decides this is OK to count to 1001 instead of 1000, just for today. But the AWS group only wants to count to 1000 because counting to 1001 is a Very Bad Thing. However, the AWS group still has to cover 1001 km. 3. The AWS group comes up with the ingenious idea of just making each "kilometer" 1.001 real-world kilometers, just for today. Thus, they will only count to 1000, but each time travel 1.001 km. The end result is the same - they will have covered 1001 km.