11 ms·
One second per second is harder than it sounds
- ars 12y agoI had an S5 motherboard (ASRock P55 Delux), after pulling out my hair I finally gave up on it and bought a new motherboard. And I made a resolution to never ever buy the V1 of anything. (It was a brand new Intel chipset, and a brand new motherboard design to go with it.) If everyone did like me and never bought V1 then there would never be a V2, I don't have a solution for that, but I refuse to be the guinea pig.
- makomk 12y agoI had a desktop PC a few years back that exhibited S5. Can't remember the exact details, but I think it turned out that Linux was using a time source which wasn't stable on that particular hardware and switching to a different time source fixed it. There's been a lot of issues like that in the past, most of which are thankfully no more.
- spydum 12y agoSince the advent of virtualization (most notoriously VMware ESX) time keeping has gotten to be much more difficult. I know back in my unix admin days, I spent more than a few days on problematic Linux kernels on VMs under ESX (made far worse when hosts were over provisioned and VMs were starved for CPU readiness). I know later enhancements like tick-less kernels may have helped, but I do not envy anyone who has to wrestle these beasts.
- joevandyk 12y agoEvery so often, my linux vm running on VMware fusion gets really out of date. I tried the "sync time" feature in VMware, didn't fix it. Eventually I had ntpdate running every minute via cron. Pretty annoying.
- isomorphic 12y agoI run various production systems in both VMware ESX and VirtualBox, and I have time issues in all VMs, regardless of hypervisor vendor. The VM tools are installed and operating "properly." The host machine clocks are synced correctly with ntp. Yet the VMs will sometimes get tens of minutes out of alignment in short order. At this point I'm thinking I need to run ntp on the physical hardware with the least jitter, then just run ntpdate in a cron job on all the VMs. This would work better than ntp clients in the VMs or the VM tools.
- ghshephard 12y agoI spent about 2+ weeks in early 2003/2004 trying to get VMware Server to properly keep the time in Sync. I had memorized the VMware whitepaper on the topic, and had followed every possible bit of advice they offered (including not using NTPD) - nothing worked. Eventually just cronned ntpdate to run every minute and that resolved the issue on all our systems. This, by the way, horrified all our NTPD theoreticians, who made it clear that running ntpdate was going to cause catastrophic things to occur in our Operations environment - but, given that our logs were all getting progressively more and more useless as we were unable to correlate times for events between servers - the worst case scenario (in my mind) had already occurred. I don't recall any particularly negative side effects as a result of our ntpdate sledgehammer. I presume things have gotten better in the last 10 years with VMware.
- bradknowles 12y agoSomewhat. Years ago, the answer was to run NTP on the host and use VMWare Tools to help ensure that good quality time was served to the clients. That's when the earlier versions of the page at <http://support.ntp.org/bin/view/Support/KnownOsIssues#Section_9.2.2.> http://support.ntp.org/bin/view/Support/KnownOsIssues#Sectio... were written. Since then, that advice has apparently been recanted. The ultimate problem with any virtualization system is the frequency and accuracy of clock updates to the client, and the loss of clock interrupts. To the degree you can solve that problem, protocols like NTP can work reasonably well to monitor and correct for various types of known clock issues.
- MichaelGG 12y agoWindows time sync only cares about being accurate to 5 !minutes, for Kerberos. HyperV and makes Windows not even able to be that accurate. Oddly enough, Linux guests have no trouble with subsecond accuracy. I'm told the w32time codebase is not something MS employees like to look at.
- 72deluxe 12y agoI don't think that w32time is something that users like to look at! I recall a situation where you run w32time on a server and it runs fine and doesn't report an error, but still silently ignore the option you set as it is using another time source from another machine that it has designated a master. I could be wrong, but I was disappointed that the utility didn't return any errors when it knew that it was going to ignore you.
- deleted 12y ago[deleted]
- dm2 12y ago28 people killed because a clock was off by .34 seconds (system hadn't been restarted in 100 hours) http://fas.org/spp/starwars/gao/im92026.htm http://fas.org/spp/starwars/gao/im92026.htm
- kranner 12y agoHaving followed the link, I'd like to mention that a Scud missile was also involved.
- lttlrck 12y agoAnd a war.
- manojlds 12y agoIsn't it actually "28 lives couldn't be saved" because of the clock issue. Also, the article states that the defense was never tried against SCUD and could have failed nevertheless.
- oakwhiz 12y agoI'm guessing that the people wouldn't have been standing within range of missile attacks if they didn't expect that the missile defense system would protect them.
- x1798DE 12y agoAlmost everywhere on earth is currently "within range of missile attacks" without any reliable missile defense shield protecting us. From the report, this was during Operation Desert Storm and they were in an Army barracks. Given that these systems had never been used to defend against SCUD missiles before, it seems likely that they were taking a calculated risk by being there, not simply assuming the missile shield would protect them.
- oakwhiz 12y ago
- chubot 12y agoI think the question is: when does it matter? What applications rely on an accurate clock? I guess it would matter if you are running some algorithm on logs that uses time as an input. Interested to hear any other examples. DJB has a clockspeed package that addresses this problem. Anyone used it? http://thedjbway.b0llix.net/clockspeed.html http://thedjbway.b0llix.net/clockspeed.html It seems interesting but I haven't (yet) run into the need for super accurate clocks (or perhaps I am not running enough machines).
- jfoutz 12y agospanner? https://en.wikipedia.org/wiki/Spanner_(database) https://en.wikipedia.org/wiki/Spanner_(database)
- dsymonds 12y agoSeveral security protocols (e.g. Kerberos) depend on systems having a somewhat reliable clock. They'll issue some sort of authentication token on system A that a user is meant to present to system B; the token has some expiry time, and the security of the system depends on systems A and B having comparable clocks.
- tmarthal 12y agoI know that offhand gaming and financial services rely on very accurate clocks at the application level. And you can't necessarily rely on the clients to tell you what their time is, as it can be spoofed.
- Arelius 12y agoGaming is actually a really bad example since as you say, you can't really on the clients telling you a time as it may be spoofed. Thus the clients times are always ignored, and thus time sync is irrelevant. I'll say that in about 10 years of working on multi-player games, I've never once put a wall, or other absolute time on the wire for any purpose than just printing server time to the player for informational purposes.
- 12y ago
- jchonphoenix 12y agoUm... In large scale systems... vector clocks? Or am I not reading this correctly (I skimmed)
- MrBuddyCasino 12y agoNot sure why downvoted, vector clocks are an interesting alternative. Its just that they won't fix your log timestamps or kerberos clock skew errors.
- kijin 12y agoIn the age of mobile phones that can get the time directly from GPS satellites with sub-100ns accuracy, the isolated hardware clock of a typical server strikes me as an arcane method of keeping time. Is there an expansion card or USB device that can get the time from GPS and feed it to something like ntpd (and adjust for leap seconds from data downloded from other networks)? Or would that be impractical because of all the walls and metal cages that surround the typical server? What about an antenna on the roof of the datacenter that supplies GPS time to every server in the building?
- rachelbythebay 12y agoDatacenter GPS-backed time sources exist. You can get little boxes which sit in a rack and hook to an antenna and feed nice time to the rest of the place. Those other machines pick it up with ntpd. The problem is when your system clock is so far out of whack (and/or unpredictable) that even ntpd won't help you. Get enough supposedly-identical machines running and this will probably bite you too.
- kastnerkyle 12y agoThis is 100% true. If you have ever watched ntpd ignore a stratum 0 clock (and every other time source) because some stratum 6 device reports the wrong time, you know the pain. Yes, this isn't supposed to happen, and yes it has happened to me many times.
- feld 12y agowell, you can override ntpd to always sync no matter how far out of whack but it's not recommended tinker panic 0
- x1798DE 12y ago>In the age of mobile phones that can get the time directly from GPS satellites with sub-100ns accuracy, the isolated hardware clock of a typical server strikes me as an arcane method of keeping time. Genuinely curious, is it actually true that you can get sub-100ns time accuracy on a mobile phone? Is this because you have a direct connection with the satellite, which is calculating where you are? I would have assumed that the uncertainty on the latency of any such connection would be higher than 100 ns.
- ChuckMcM 12y agoFor a long time synchronized clocks was the networking code's equivalent joke to Regular Expressions ("Now you've got two problems ...") That said, having implemented a 100ns accurate clock using the PPS on a $30 GPS module (Adafruit) and a Beaglebone Black (multiple) these days there are some interesting alternatives. It would be interesting if the Open Compute folks would specify a 1PPS GPIO input as part of the spec, you could easily deliver rack level 1PPS accurate signals throughout the data center pretty cost effectively (a few wires, a transmitter, a TDR tool. Not that it will ever be a 'solved' problem but it certainly makes some transactional protocols safer.
- rdtsc 12y agoI've used PTP with special timestamping hardware (network cards) and could synchronize machines on a LAN pretty well. Now that doesn't guarantee global sync just sync between machines. GPS NTP server can be obtained for as low as $600 or so. So those are becoming not as exotic now.
- nick_riviera 12y agoWe had radio time receivers (Rugby MSF, now Anthorn MSF) for a long time in the UK. Just plug them into an RS232 hole and have a site-wide master NTP server on that box, fire up the daemon and you're done. I think the last one we paid for was in the region of $250 as the hardware is very simple. http://www.npl.co.uk/science-technology/time-frequency/products-and-services/time/msf-radio-time-signal http://www.npl.co.uk/science-technology/time-frequency/produ... GPS is an interesting prospect but they are pretty HF which is a PITA if you have a unit inside a rack inside a machine room inside a building. MSF is 60KHz so is ground wave propagation so it had great coverage.
- easytiger 12y agoBut worth noting it is only ±1 ms accuracy which is not useful in many applications.
- 12y ago
- rdtsc 12y agoYou can buy GPS receivers and install antennas. Those are pretty good but expensive. For S2, it can be configured to never jump. Instead you can just the time manually (run the equivalent of 'ntpdate <server>') at boot perhaps. Then don't start your software until ntp has been synchronized. Then maintain a watchdog that will alert you if you drop to a too low of a stratum. Don't have an answer for S4 and S5. But I have heard that chrony is a good replacement for ntp (same network protocol but different discipline, faster conversion). http://chrony.tuxfamily.org/ http://chrony.tuxfamily.org/ For a whole new other thing there is PTP (https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol). This is good if you want to synchronize machines to each other to withing a couple of hundred micros (maybe more with special network cards and switches).
- kastnerkyle 12y agoThe question is - how often do you reboot? For long running high uptime applications "ntpdate at boot" doesn't cut it... and the other solutions are nasty. In my experience, ntpd will ignore clocks almost at a whim even if they are stratum 0! Some stratum 6 clock decides to report a different time, and suddenly all your other sources are ignored until an ntpd restart. If your clock drifted too much... even ntpd restart with options to "force the time" can fail if the clock is too far out of whack. The you are back to forcing ntpdate - hope none of your processes are time dependent when system time jumps by 3 or 4 seconds. Half of the GPS PCI devices require you to hack ntpd directly to get anything working, and the Ethernet sources have a strange tendency to go belly up after a year or 2 of running on non-120V sources. Not to mention that if you run the antenna cable too long, you might not notice that your GPS amplitudes are a little low until the next cloudy day. Or if someone adds a new hunk of metal on your roof, or blocks line of sight to some portion of the sky... Long story short, time is a nightmare and I am glad other people think so too.
- onli 12y agoThat is the nice thing about chrony. It tries to synchronize clocks even when offline, it works not only on a reboot. If it works as advertized (and I tried it some years ago at home and it worked there, for what it's worth), it should fix all of the issues described in the article.
- lamontcg 12y agoRun a cronjob every hour which: a. ensures ntpd is running and starts it if its not. b. checks `nptq -rv` and looks for the self-reported state of ntp and bounces if it is not sync'd. c. 'manually' checks ntpdate for the drift against a known stratum 1 and bounces the daemon if the drift is out of bounds (something like 50ms is a pretty good cutoff since ntpd normally does much better than that). You can run this out of chef/puppet/whatever but it needs to run with a frequency of about an hour (faster than that and ntpd often won't settle well, and slower than that and you can be needlessly waiting too long to fix problems). Running it as a cronjob uncouples it from how often your config management runs. This will catch all kinds of issues -- crashing ntpds, ntdps that randomly lose sync, kernel problems that keep them from keeping sync, ntpds that lie about their sync status, etc. I set it up to e-mail me as a form of monitoring (with heavy-duty procmail filters). Had 30,000 servers with this script running against it. I think 6 of them had shitty clocks that were getting reset every single hour, but were still keeping time correctly to under a second. Every couple days there would be a server that'd flip out and send a few e-mails until it quieted down and synch'd. At one point we had a shipment of several hundred new servers that all had issues with kernel drivers and the cronjob was bouncing ntpd constantly on them until a kernel upgrade fixed the problem. I also used to bounce ntpd once a night, but had to take that out because it would cause non-monotonic slew in the clock which timers around service calls would turn up as negative seconds that due to unsigned int conversion would turn into 4 billion second p100 times. You do want to be careful about things like network outages, if you can't ping your upstream stratum1/2 its probably better to leave things as it is. And again, you really want to limit the frequency that you bounce it at, and you want to be aware of servers where it gets into a state where you're bouncing ntpd constantly.
- easytiger 12y agoThis is a terrible idea when dealing with an application other than "users checking the time on their system clock". Performing hard resets of the clock continually is a catastrophic infrastructure failure. A high performance 24/7 application that relies on sub millisecond let alone sub microsecond accuracy across multiple physical systems will be utterly destroyed by this approach.
- easytiger 12y agoErr no mention of PTP at all? There are hardly any oscillators which can be plugged into a host which have any decent holdover. One should never expect them to run in reasonable time. Standard TCXO prob losses 100 ms a day. Rubidium loses 0.01 ms a day. Time providers include the perceived accuracy of the clock in the protocol so that the client can make a decision about which time source to use. Relying on the oscillator is never done in time critical applications except in emergency states where GPS is out of service. Of course a lot of people run relative clock domains, which is just a pretty bad idea for most applications.
- socialist_coder 12y agoI have a problem in my AWS Windows cluster with the clocks as well. We have a mobile game with a tournaments feature, and tournaments can be as short as 5 minutes. So, it's important that all the clients get the same "end time". Normally, at least 10% of the VMs are between 5 and 20 seconds off from the average. By default, the VMs are syncing their time to an NTP server, but it just doesn't seem to be enough. I haven't changed any settings as of yet. What are some of the things I could do on Windows box to sync up the clocks in my cluster? Scheduled shell scripts to force more clock syncs? Or maybe something on the application level?
- bradknowles 12y agoFirst off, Windows isn't the greatest client for trying to get good timesync. The way the OS usually handles interrupts isn't sufficiently precise and accurate. That said, it's not uncommon for real-hardware Windows clients to get decent timesync down into the double-digit millisecond range -- usually about ~40-50ms or so. Second, you're trying to run them in a virtualization environment, and that plays holy hell with whatever good timesync you might otherwise have been able to get. And AWS is probably about as bad as you could reasonably get in this regard -- VMWare has had a long time to get to where they are with regards to timesync for client OSes, and that's not particularly good. So, your best bet may be to run NTP clients on the Windows machines, but to set them up to run at an artificially high rate of updates. Or maybe even run ntpdate periodically against good real-hardware time servers that are available to you. You're starting off with both hands tied behind your back, and that's a hard place to come back from.
- kiram9 12y agoThere is a standard to keep time on LANs now: IEEE-1588 which is built into a lot of new network cards. It enables the NIC to time stamp incoming packets which can be used to achieve sub microsecond timing. Normally it can be implemented with a single GPS receiver on the network that then feeds all servers/devices. It is starting to become very common in broadcast applications as a reference. Furthermore most manufactures already offer some application notes/libraries to make use of it. And linux offers PTPd which implements the standard. http://ptpd.sourceforge.net/ http://ptpd.sourceforge.net/ http://en.wikipedia.org/wiki/Precision_Time_Protocol http://en.wikipedia.org/wiki/Precision_Time_Protocol http://www.nist.gov/el/isd/ieee/ieee1588.cfm http://www.nist.gov/el/isd/ieee/ieee1588.cfm
- bradknowles 12y agoPTP is a good choice for synchronizing clocks on a per-segment basis, but it doesn't work across a WAN and there are lots and lots of devices out there that don't support it. NTP is good for doing time sync across the WAN, but to get the most out of it for certain applications you need to hook it up to reference clocks. So, NTP can use PTP as a reference clock, and Bob's your uncle.
- feld 12y agoFreeBSD timecounter in detail: http://phk.freebsd.dk/pubs/timecounter.pdf http://phk.freebsd.dk/pubs/timecounter.pdf