9 ms·
NIST was 5 μs off UTC after last week's power cut
- asdfman123 10mo agoI find this topic and thread fascinating. I took too much Adderall today.
- qmr 10mo agoGah, just when you think you can trust time.nist.gov Suggestions from the community for more reliable alternatives?
- evanriley 10mo ago> Gah, just when you think you can trust time.nist.gov You still can... If you're that considered about 5 microseconds: Build your own Stratum 1 time server https://github.com/geerlingguy/time-pi https://github.com/geerlingguy/time-pi or just use ntppool https://www.ntppool.org/en/ https://www.ntppool.org/en/
- eddyg 10mo agoBe aware that there are members of the NTP pool with less-than-honorable intentions and you don't get to pick-and-choose. Yes, they all should provide the time, but they also get your IP address. For example: unlike the IPv4 space, the IPv6 space is too big too scan, so a number of "researchers" (if you want to call them that) put v6-capable NTP servers in the NTP pool to gather information about active v6 blocks to scan/target.
- beala 10mo agoIt sounds like GPS, and thus a GPS-based stratum 1 server, uses these time servers, but they were successfully failed over: > Jeff finished off the email mentioning the US GPS system failed over successfully to the WWV-Ft. Collins campus. So again, for almost everyone, there was zero issue, and the redundancy designed into the system worked like it's supposed to. So failures in these systems are potentially correlated. The author mentions another solution. Apparently he runs his own atomic clock. I didn’t know this was a thing an individual could do. > But even with multiple time sources, some places need more. I have two Rubidium atomic clocks in my studio, including the one inside a fancy GPS Disciplined Oscillator (GPSDO). That's good for holdover. Even if someone were jamming my signal, or my GPS antenna broke, I could keep my time accurate to nanoseconds for a while, and milliseconds for months. That'd be good enough for me.
- geerlingguy 10mo agoThe CSACs that I have in a couple devices are 'atomic', and use Rubidium, but they're a bit lower accuracy than Cesium clocks [1] or Hydrogen Masers [2]. There are a few folks on the time-nuts mailing list who own such exotic pieces of hardware, but those are pretty far out of reach for most! [1] https://www.microchip.com/en-us/products/clock-and-timing/components/atomic-clocks/atomic-system-clocks/cesium-time/5071a https://www.microchip.com/en-us/products/clock-and-timing/co... [2] https://www.microchip.com/en-us/products/clock-and-timing/components/atomic-clocks/atomic-system-clocks/mhm-2020-hydrogen-masers https://www.microchip.com/en-us/products/clock-and-timing/co...
- rcxdude 10mo agoAtomic clocks cover a pretty big range of performance nowadays. You can pick up a used but serviceable rubidium frequency reference for a few hundred dollars but the difference between it and the top of the line clocks is almost as big as the difference between a it and a good pendulum clock.
- ajkjk 10mo agotheir handling it responsibly seems like more evidence for trusting them, not less?
- monster_truck 10mo agoI'm more concerned about what you think they did to earn your trust in the first place
- ianburrell 10mo agoMost places that need accurate time get it from GPS. That is 10-100 ns. Also, you can use multiple NIST servers. They have ones in Fort Collins, CO and Gaithersburg, MD. Most places shouldn't use NIST directly but Stratum 1 name servers. Finally, NTP isn't accurate enough, 10-100 ms, for microsecond error to matter.
- vel0city 10mo agoUse the other servers as well: https://tf.nist.gov/tf-cgi/servers.cgi https://tf.nist.gov/tf-cgi/servers.cgi For instance, time-a-wwv.nist.gov. One should configure a number of different NTP sources instead of just a single host.
- ssl-3 10mo agoYes. Use NTP with ≥4 diverse time sources, just as RFC 5905 suggests doing. And use GPS. (If you're reliant upon only one source of a thing, and that thing is important to you in some valuable way, then you're doing it wrong. In other words: Backups, backups, backups.)
- ChrisArchitect 10mo agoMore discussion: NTP at NIST Boulder Has Lost Power https://news.ycombinator.com/item?id=46334299 https://news.ycombinator.com/item?id=46334299
- gnabgib 10mo agoFrom NPR (22 points) https://news.ycombinator.com/item?id=46351105 https://news.ycombinator.com/item?id=46351105
- politelemon 10mo agoI'm missing the nuance or perhaps the difference between the first scenario where sending inaccurate time was worse than sending no time, versus the present where they are sending inaccurate time. Sorry if it's obvious.
- BuildTheRobots 10mo agoIt's a good question, and I wondered the same. I don't know, but I'd postulate: As it stands at the minute, the clocks are a mere 5 microseconds out and will slowly get better over time. This isn't even in the error measurement range and so they know it's not going to have a major effect on anything. When the event started and they lost power and access to the site, they also lost their management access to the clocks as well. At this point they don't know how wrong the clocks are, or how more wrong they're going to get. If someone restores power to the campus, the clocks are going to be online (all the switches and routers connecting them to the internet suddenly boot up), before they've had a chance to get admin control back. If something happened when they were offline and the clocks drifted significantly, then when they came online half the world might decide to believe them and suddenly step change to follow them. This could cause absolute havoc. Potentially safer to scram something than have it come back online in an unknown state, especially if (lots of) other things are are going to react to it. In the last NIST post, someone linked to The Time Rift of 2100: How We lost the Future --- and Gained the Past. It's a short story that highlights some of the dangers of fractured time in a world that uses high precision timing to let things talk to each other: https://tech.slashdot.org/comments.pl?sid=7132077&cid=49308245 https://tech.slashdot.org/comments.pl?sid=7132077&cid=493082...
- opello 10mo agoThe 5us inaccuracy is basically irrelevant to NTP users, from the second update to the Internet Time Service mailing list[1]: To put a deviation of a few microseconds in context, the NIST time scale usually performs about five thousand times better than this at the nanosecond scale by composing a special statistical average of many clocks. Such precision is important for scientific applications, telecommunications, critical infrastructure, and integrity monitoring of positioning systems. But this precision is not achievable with time transfer over the public Internet; uncertainties on the order of 1 millisecond (one thousandth of one second) are more typical due to asymmetry and fluctuations in packet delay. [1] https://groups.google.com/a/list.nist.gov/g/internet-time-service/c/OHOO_1OYjLY https://groups.google.com/a/list.nist.gov/g/internet-time-se...
- V__ 10mo agoHas anyone here ever needed microsecond precision? Would love to hear about it.
- marcosdumay 10mo agoYou probably want to ask about accuracy. Any random microcontroller from the 90s needs microsecond precision.
- hnuser123456 10mo agoThe high frequency trading guys edit: also the linked slides in TFA
- andrewxdiamond 10mo agoWe use nanosecond precision driven by GPS clocks. That timestamp in conjunction with star tracker systems gives us reliable positioning information for orbital entities. https://en.wikipedia.org/wiki/Star_tracker https://en.wikipedia.org/wiki/Star_tracker
- loeg 10mo agoI mean, we routinely benchmark things that take microseconds or less. I've seen a 300 picosecond microbenchmark (single cycle at 3GHz). No requirement that absolute time is correct, though.
- IceWreck 10mo agoWe need nanosecond precision for trading - basically timestamping exchange/own/other events and to measure latency.
- esseph 10mo agoTelecom. Precision Time Protocol gets you sub-microsecond. https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol
- sgillen 10mo agoWe don't use NTP, but for robotics, stereo camera synchronization we often want the two frames to be within ~10us of eachother. For sensor fusion we then also need a lidar on PTP time to be translated to the same clock domain as cameras, for which we also need <~10us. We actually disable NTP entirely (run it once per day or at boot) to avoid clocks jumping while recording data.
- geetee 10mo agoNow I'm curious... How the hell do you synchronize clocks to such an extreme accuracy? Anybody have a good resource before I try to find one myself?
- mmmlinux 10mo agoAre there any plans being made to prevent this happening in the future?
- kibwen 10mo agoYes, the US government is banning all those democrat windmills that conspired to blow over the NTP server.
- deleted 10mo ago[deleted]
- groundzeros2015 10mo ago??? The power outage was voluntary and surrounding towns chose not to turn off power. They could absolutely make infrastructure changes, and I’m sure the backups for power could make changes too.
- Muromec 10mo agoIt would greatly decrease electricity prices too. God knows how expensive it is to power those gigantic spinning things
- ykonstant 10mo agoI laughed audibly at that!
- Topgamer7 10mo agoOut of curiosity, can anyone say the most impactful things they've needed incredibly accurate time for?
- srean 10mo agoNot sure but synthetic massive aperture radio telescope would need syncing their local clocks. I defer to the experts.
- jasonwatkinspdx 10mo agoAs far as I'm aware they just timestamp the sample streams based on a local gps backed atomic reference. Then when they get the data/tapes in one computing center they can just run a more sophisticated correlation entirely in software to smooth things out.
- Sanzig 10mo agoSpacecraft state vectors.
- rcleveng 10mo agoSpanner (See https://docs.cloud.google.com/spanner/docs/true-time-external-consistency https://docs.cloud.google.com/spanner/docs/true-time-externa...)
- 0x457 10mo agoDoes it need to be this close to NIST, or just relative to each other? Because the latter one is solved by PTP.
- rcleveng 10mo agoAs far as I remember, near each other, but the comment I was replying to was specifically about needing accuracy. It's been over a decade now since I managed the truetime team at google, things may have changed since :)
- 10mo ago
- ziml77 10mo agoNitpick: UTC stands for Coordinated Universal Time. The ordering of the letters was chosen to not match the English or the French names so neither language got preference.
- stavros 10mo agoIt also stands for Universel Temps Coordonné.
- Aloisius 10mo agoIt's le temps universel coordonné in French.
- userbinator 10mo agoUniversal Time, Coordinated.
- ambicapter 10mo agoThat's an interesting rationale.
- hunter2_ 10mo agoReminds me of when a group is divided into two parts, dubbed group 1 and group A, such that neither feels secondary.
- semenko 10mo agoI found the most interesting part of the NIST outage post [1] is NIST's special Time Over Fiber (TOF) program [2] that "provides high-precision time transfer by other service arrangements; some direct fiber-optic links were affected and users will be contacted separately." I've never heard of this! Very cool service, presumably for … quant / HFT / finance firms (maybe for compliance with FINRA Rule 4590 [3])? Telecom providers synchronizing 5G clocks for time-division duplexing [4]? Google/hyperscalers as input to Spanner or other global databases? Seriously fascinating to me -- who would be a commercial consumer of NIST TOF? [1] https://groups.google.com/a/list.nist.gov/g/internet-time-service/c/OHOO_1OYjLY https://groups.google.com/a/list.nist.gov/g/internet-time-se... [2] https://www.nist.gov/pml/time-and-frequency-division/time-services/time-over-fiber https://www.nist.gov/pml/time-and-frequency-division/time-se... [3] https://www.finra.org/rules-guidance/rulebooks/finra-rules/4590 https://www.finra.org/rules-guidance/rulebooks/finra-rules/4... [4] https://www.ericsson.com/en/blog/2019/8/what-you-need-to-know-about-timing-and-sync-in-5g-transport-networks https://www.ericsson.com/en/blog/2019/8/what-you-need-to-kno...
- dmurray 10mo agoI never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems. Sub-microsecond latency, yes, but when that mattered it was in order to do something as soon as possible rather than as close as possible to Wall Clock Time X. Still useful for post-trade analysis; perhaps you can determine that a competitor now has a faster connection than you. The regulatory requirement you linked (and other typical requirements from regulators) allows a tolerance of one second, so it doesn't call for this kind of technology.
- blibble 10mo ago> I never saw a need for this in HFT. In my experience, GPS was used instead, but there was never any critical need for microsecond accuracy in live systems. mifid ii (uk/eu) minimum is 1us granularity https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2017.087.01.0148.01.ENG&toc=OJ:L:2017:087:TOC https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:...
- deleted 10mo ago[deleted]
- loph 10mo agoOnly Boulder servers lost sync. To say NIST was off is clickbait hyperbole. This page: https://tf.nist.gov/tf-cgi/servers.cgi https://tf.nist.gov/tf-cgi/servers.cgi shows that NIST has > 16 NTP servers on IPv4, of those, 5 are in Boulder and were affected by the power failure. The rest were fine. However, most entities should not be using these top-level servers anyway, so this should have been a problem for exactly nobody. IMHO, most applications should use pool.ntp.org
- NetMageSCW 10mo agoWho does use those top-level servers? Aren’t some of them propagating the error or are all secondary level servers configured to use dispersed top-level servers? And how do they decide who is right when they don’t match? Is pool.ntp.org dispersed across possible interference and error correlation?
- mcpherrinm 10mo agoYou can look at who the "Stratum 2" servers are, in the NTP.org pool and otherwise. Those are servers who sync from Stratum 1, like NIST. Anyone can join the NTP.org pool so it's hard to make blanket statements about it. I believe there's some monitoring of servers in the pool but I don't know the details. For example, Ubuntu systems point to their Stratum 2 timeservers by default, and I'd have to imagine that NIST is probably one of their upstreams. An NTP server usually has multiple upstream sources and can steer its clock to minimize the error across multiple servers, as well as detecting misbehaving servers and reject them ("Falseticker"). Different NTP server implementations might do this a bit differently.
- neonnomad 10mo agoI've been running servers for the pool for years. They are checked regularly for accuracy/uptime or it's score goes down in the pool and eventually gets removed. I sync from 5 stratum 1 servers and use chrony these days. Facebook had a really interesting engineering blog about building their own timeservers: https://engineering.fb.com/2020/03/18/production-engineering/ntp-service/ https://engineering.fb.com/2020/03/18/production-engineering... Really well written for anyone who is interested.
- ComputerGuru 10mo agoNot exactly the topic of discussion but also not not on topic: just wanted to sing praise for chrony which has performed better than the traditional os-native NTP clients in our testing on a myriad of real and virtualized hardware.
- steve1977 10mo agoChrony is the default already in some distros (RHEL and SLES that I know of), probably for this very reason.
- voidUpdate 10mo agoMaybe I missed something, but I don't quite understand the video title "NIST's NTP clock was microseconds from disaster". Is there some limit of drift before it's unrecoverable? Can't they just pull the correct time from the other campus if it gets too far off?
- nottorp 10mo agoI'll consider it clickbait... ... unless someone with real experience needing those tolerances chimes in and explains why it's true.
- voidUpdate 10mo agoI'd have thought Jeff to be above clickbait, but here we are
- nottorp 10mo agoThe web article title doesn't mention any disaster. I don't see the embedded youtube video - probably have to many privacy options on.
- mrguyorama 10mo agoIf Jeff wants to maintain his job of making Youtube videos, youtube itself forces him to make clickbait thumbnails and titles. They have A/B testing infrastructure available to creators for exactly that purpose. If you say "Oh but he doesn't have to use that" you are wrong. If a creator's videos get low enough click through and "engagement", youtube just stops showing it to people. Jeff, like every creator on youtube, is in an antagonistic deathmatch for your attention with people like Mr. Beast, and Youtube would rather five Mr Beasts than a thousand Jeffs. Youtube builds their platform to empower and enrich Mr Beast, and to force everyone else to adopt more of Mr Beasts methods to stay getting views, and getting their paycheck It does not matter if you are "subscribed" to a channel for example. Youtube will still fail to show you new videos from your subscriptions if you don't play the stupid game enough. Stop blaming the people who did not at all make this choice. Blame Youtube. Support platforms, like nebula and whatever that guntube service is called, with real money. Did you know this video existed before it was posted to HN? I did, because it showed up in my feed even though I am not subscribed to jeff, because he plays the game well. I watched it too because I had no idea that weather was causing such a problem in Colorado and love hearing about NIST and the people who basically run the infrastructure of the internet that everyone takes for granted.
- meindnoch 10mo agoI know some HFT people who made a few hundred K off of this.
- tiborsaas 10mo agoCan you elaborate on how? Did they consciously exploit this or their systems just had a lucky glitch?