8 ms·
The Limits of NTP Accuracy on Linux
- nullc 1y agoGPS timing modules should have a sawtooth correction value that will tell you the error between the upcoming pulse and GPS time. The issue is that PPS pulse has to be aligned to the receiver's clock. Using that will remove the main source of jitter.
- RossBencina 1y agoOnly the expensive ones have the correction capability (e.g. uBlox LEA-M8T) hat tip to tverbeure: https://news.ycombinator.com/item?id=44147523 https://news.ycombinator.com/item?id=44147523 Aligning the PPS pulse with an asynchronous local clock is going to require a very large number of measurements, or a high resolution timer (e.g. a time to digital converter, TDOA chip, etc. there are a few options.)
- wowczarek 1y agoTo an extent. You can get previous generation GNSS receivers with sawtooth correction for cheap, eBay is full of those, say an old Trimble Resolution whatever, and lots of LEA-6T carrier boards going for $20-range, and bare modules for less. I would trust those carrier boards more though, less chance of getting a fake module.
- nullc 1y agoThe timing receivers often have other advantages beyond just sawtooth, including stuff like being able to produce a time pulse with a single working satellite in view once their position is learned.
- wowczarek 1y agoShould have clarified: previous generation timing receivers. Yes, self-survey + stationary mode + O/D mode where you can track a single SV and such, are essential for stable time sync, not just sawtooth. There is an exception re. sawtooth, but only a recent one: the Furuno GT-100, priced between the ZED-F9T and the Mosaic-T, has 200 ps clock resolution and doesn't even provide a quantisation error output.
- RossBencina 1y agoGreat, thanks. For anyone following along at home, LEA-6T reports the time pulse error in picoseconds in the TIM-TP message. See "11 Timepulse" on page 26, and/or search for "TIM-TP" in the product spec: https://content.u-blox.com/sites/default/files/products/documents/u-blox6_ReceiverDescrProtSpec_%28GPS.G6-SW-10018%29_Public.pdf https://content.u-blox.com/sites/default/files/products/docu... There are cheap LEA-8MT on eBay too, especially if you have a method to pull the module off a scrap board fragment.
- wowczarek 1y agoApplying sawtooth correction will remove _some_ of the jitter but at the time server level, not network level, and quantisation error is not the main source of jitter here (I'm talking long time constants) - Packet Delay Variation (PDV) and internal time comparisons are. Plus, any decent loop should average the sawtooth and transform it into fluctuations slow enough that they will not have that much effect on what is being measured in the blog post - the output of the time server looks nothing like the raw 1PPS input, at least in the short term it doesn't. Of course sawtooth should be removed and let's hope his time servers do it, especially the RPi ones.
- nullc 1y ago> any decent loop should average the sawtooth You can't really, depending on their relative phases and the resulting aliasing products the average of the sawtooth error can still have an arbitrary offset which last for an arbitrarily long time. > that they will not have that much effect okay fine for some definition of 'not much' that's true. But failing to account for it can result in a bigger error than many people expect-- and in an annoying way, since when you test it might be in a state where it is averaging out okay but then later shift into a state where it's producing an offset that doesn't average out. Assuming your receiver outputs the correction it's pretty easy to handle, so long as you know it's a thing.
- watersb 1y agoSegal's Law: "A man with a watch knows what time it is. A man with two watches is never sure." https://en.m.wikipedia.org/wiki/Segal's_law https://en.m.wikipedia.org/wiki/Segal's_law
- JdeBP 1y agoA person with two watches finds xyrself suddenly in the messy business of doing full NTP, rather than the much simpler model of SNTP. (-:
- nullc 1y agoA person with three or more watches knows what time it is in proportion to the square root of the number of watches.
- stinkbeetle 1y agoA person with four watches knows what time it is in proportion to 2?
- dspillett 1y agoSort of. With two watches your confidence is ~1.4 of an arbitrary measurement, with three it is ~1.7, with 4 ~2, etc. Though this is an ever-growing sequence. A better model might be to measure the confidence with something like (x-1)/x as this grows, more slowly with each step, towards 1, without really getting there until infinity. With two watches you are 50% of maximum confidence in your time, with three 66%, with four 75%, five->80%, and so on.
- ofalkaed 1y agoIf you actually care about what time it is you need at least three so you can average them and knock out the error. The Beagle carried 22 when it also carried Darwin, in the nearly 5 year expedition they only lost 30 odd seconds.
- azalemeth 1y agoMy experience with rt Linux is that it can be exceptionally good at keeping time, if you give up the rest of the multitasking micro sleeping architecture. What do you need this accurate time for? I'm equally sure, as acknowledged, the multipath routing isn't helping either.
- mustache_kimono 1y ago> What do you need this accurate time for? Securities regulation?: https://podcasts.apple.com/us/podcast/signals-and-threads/id1528917129?i=1000494740692 https://podcasts.apple.com/us/podcast/signals-and-threads/id...
- JdeBP 1y agoBear in mind that the author specifically reminds us, halfway down, that the goal is consistency, not accuracy per se. Making all of the systems accurate to GNSS is merely a means of achieving the consistency goal so that event logs from multiple systems can be aligned.
- mschuster91 1y ago> What do you need this accurate time for? Say you are running a few geographically apart radio receivers to triangulate signals, you want to have all of them as closely synchronized as possible for better accuracy.
- michaelt 1y ago> What do you need this accurate time for? Some major uses of high-precision timing, albeit not with NTP, include: * Synchronising cell phone towers, the system partly relies on precise timing to avoid them interfering with one another. * Timestamping required by regulators, in industries like high-frequency trading. * Certain video production systems, where a ten-parts-per-million framerate error would build up into an unacceptable 1.7 second error over 48 hours. * Certain databases (most famously Google Spanner) that use precise timing to ensure events are ordered correctly * As a debugging luxury in distributed systems. In some of these applications you could get away with a precise-but-free-floating time, as you only need things precisely synchronised relative to one another, not globally. But if you're building an entire data centre, a sub-$1000 GPS-controlled clock is barely noticeable.
- sugarpimpdorsey 1y agoThere are so many inaccurate technical details here I wouldn't know where to begin, let alone write a blog post. Sigh.
- jmpman 1y agoI would be extremely interested in reading your blog post. Fascinating topic.
- ainiriand 1y agoUnfortunately I think the same as you. The provided details in the blog post are by no means any way of doing any sort of time benchmark or network i/o benchmark. For starters, he is comparing times from tsc enabled hardware (x86_i64), with raspberry pi which are arm. Network i/o benchmarking on linux should be done with system calls to the network cards or input devices and not through the kernel drivers etc...
- Avamander 1y ago> For starters, he is comparing times from tsc enabled hardware (x86_i64), with raspberry pi which are arm. Well, that TSC-enabled hardware also has other peripherals (like SMBUS as mentioned in the article) that on the other hand introduce errors into the system. I personally use a RPi4 with its external oscillators replaced with a TXCO. Some sellers on AliExpress even have kits for "audiophiles" that let you do this. It significantly improved clock stability and holdover. So much so that "chronyc tracking" doesn't show enough decimal places to display frequency error or skew. It's unfortunate though that the NIC does not do timestamping. (My modifications are similar to these: https://raspberrypi.stackexchange.com/a/109074 https://raspberrypi.stackexchange.com/a/109074) I'd love to find an alternative cheap (microcontroller-based) implementation that could beat it.
- mwpmaybe 1y agoThe CM5's NIC has timestamping, but not sure if there's a TXCO hack for it.
- 1y ago
- diarmuidc 1y agoWhy is there no mention of PTP here? If you want accurate time synchronisation in a network just use the correct tool, https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol Linux PTP (https://linuxptp.sourceforge.net/ https://linuxptp.sourceforge.net/) and hardware timestamping in the network card will get you in the sub 100ns range
- rendaw 1y agoThere's a discussion on that in the comments at the bottom of the article, where the author explains why it wasn't analyzed.
- jacob2161 1y agoChrony over NTP is capable of incredible accuracy, as shown in the post. Most users who think they need PTP actually just need Chrony and high quality NICs. Chrony is also much better software than any of the PTP daemons I tested a few years ago (for an onboard autonomous vehicle system).
- eqvinox 1y agoNTP fundamentally cannot reach the same accuracy as PTP because Ethernet switches introduce jitter due to queueing delays and can report that in PTP but not NTP.
- erincandescent 1y agoChrony can do NTP encapsulated inside PTP packets so as to combine the best parts of both protocols
- eqvinox 1y agoThat's not exactly NTP though ;) I'll also say PTP is superior since it syncs TAI rather than NTP's UTC. Which probably isn't going to change even with NTPv5.
- 1y ago
- throwawaysoxjje 1y agoIt’s wild they talk about the jitter in the pps signals but glossed over the jitter the oscilloscope?
- marshray 1y agoThe scope should be capturing samples from the three channels synchronously. It appears to be set to trigger on the bottom trace (it appears still) and then retrospectively display the other two.
- magicalhippo 1y agoGiven he's using the desktop PPS signal as a trigger reference, and compares the relative times, how would intrinsic scope jitter significantly affect that?
- deleted 1y ago[deleted]
- guenthert 1y agoNot sure about the Siglent oscilloscope used here, but my old LeCroy WaveAce 2032 (were Siglent had obviously it's hands on) has a trigger jitter of 0.4ns. I'd think the one used here will be of the same order of magnitude, i.e. negligible. Uh, the Siglent SDS 1204X-E used here has a "new, innovative digital trigger with low latency" ... But yes, as others have commented already, if only the relative jitter between the signals is of interest, the trigger jitter itself is inconsequential.
- magicalhippo 1y ago> Uh, the Siglent SDS 1204X-E used here has a "new, innovative digital trigger with low latency" ... Datasheet claims trigger jitter of <100 ps.
- guenthert 1y agoWell, I looked for the data sheet and couldn't find it, just the ominous quote. Now, newly motivated, I did eventually find it (trick is to look for the data sheet of the series, not the model; SDS1204X-E being the 200MHz, 4 channel model of the SDS1000X-E series): https://www.batronix.com/files/Siglent/Oszilloskope/SDS1000X-E/SDS1004X-E_DataSheet_EN.pdf https://www.batronix.com/files/Siglent/Oszilloskope/SDS1000X...
- lifeline82 1y ago[dead]
- klaas- 1y agomaybe I missed that, but why not just combine ptp and ntp within chrony, it does support that.
- Galanwe 1y agoSee also https://en.m.wikipedia.org/wiki/White_Rabbit_Project https://en.m.wikipedia.org/wiki/White_Rabbit_Project for this interested in PTP / low latency time sync
- contingencies 1y agoWhen it comes to realtime guarantees, bare metal code on a dedicated IC or MCU is by definition better than anything running on a general purpose OS on a general purpose CPU. Even if you tune the hell out of it, the latter will have more room for bugs, edge cases, drift, delayed powerup, supply chain idiosyncracies, etc. FYI GNSS processing chips cost $1.30 these days. https://www.lcsc.com/datasheet/C3037611.pdf https://www.lcsc.com/datasheet/C3037611.pdf
- eqvinox 1y agoGPS modules need to be put in a special stationary mode (and ideally measured-in to a location for a day or two) to get accurate timing. I'm consistently achieving ca. 10ns of deviation. Hope the author didn't forget this. (But it might also just be crappy GPS modules, I'm using u-blox M8T which are specifically intended for timing.)
- RossBencina 1y agoWhat method do you use to measure 10ns deviation?
- eqvinox 1y agoDelta between modules on a scope (offset values on the hardware timestamp on the immediately connected PTP clock also line up with this) [Caveat: everything is in the same room with the same ambient temperature drifts…]
- mwpmaybe 1y agoInteresting. The serial GPS module currently wired up to my Raspberry Pi doesn't have a stationary mode per se but there is a feature called AlwaysLocate that seems related. I can choose between Periodic Backup/Standby and AlwaysLocate Backup/Standby modes. I'll need to look into this... ETA: I can also increase the nav speed threshold to 2m/s.
- eqvinox 1y agoNot all modules have this feature; and it's also locked behind feature/license bits sometimes. It's obviously not needed for normal GPS use… u-blox timing targeted modules definitely have it. Some have a "measure-in" mode where you let it sit for a while (days) and it does all the setup automatically. Other cases you actually have to feed things into the module (annoying and error prone…) It's simply that if you know your location, you can remove that as free variable from the equations and instead constrain the time further.
- RossBencina 1y agoThere was some related discussion a couple of weeks ago here: Graham: Synchronizing Clocks by Leveraging Local Clock Properties (2022) [pdf] (usenix.org) https://news.ycombinator.com/item?id=44860832 https://news.ycombinator.com/item?id=44860832 In particular the podcast about Jane Street's NTP setup was discussed.
- wowczarek 1y agoThe problem with these types of posts is that this is an area that many are unfamiliar with (at least not in depth) and making lots of authoritative statements makes it believable at face value. There are so many variables to network time sync that you design to minimise them. For example no multipath and no asymmetry unless you have PTP P2P transparent clocks everywhere. The author also mixes precision with accuracy and relies on self-reported figures from NTP (chrony says xxx ns jitter). With every media speed change you get asymmetry which affects accuracy (not always precision though). So your 100m->1G link for example will already introduce over 1 us of error (to accuracy!), but NTP will never show you this and nothing will unless you measure both ends with 1PPS, and the only way around it is PTP BC or TC. There is a very long list of similar clarifications that can be made. For example nothing is mentioned about message rates / intervals, which are crucial for improving the numbers of samples filters work with - and the fact that ptp4l and especially phc2sys aren't great with filtering. Finally getting time into the OS clock, unless you use PCIE PTM which practically limits you to newer Intel CPUs and newer Intel NICs, relies on a PCIE transaction with unknown delays and asymmetries, and without PTM (excluding few NICs) your OS clock is nearly always 500+ ns away from the PHC and you don't know by how much and you can't measure it. It's just a complex topic and requires an end to end, leave no stone unturned, semi-scientific approach to really present things correctly.
- mlichvar 1y agoI agree with most of what you said. The author has other posts in the series where he tried to measure the accuracy relative to the PHC (not system clock) using PPS: https://scottstuff.net/posts/2025/06/02/measuring-ntp-accuracy-with-an-oscilloscope/ https://scottstuff.net/posts/2025/06/02/measuring-ntp-accura... Steering the same PHC with phc2sys as chronyd is using for HW timestamping is not the best approach as that creates a feedback loop (instability). It would be better to leave the PHC running free and just compare the sys<->PHC with PHC<->PPS offsets. > So your 100m->1G link for example will already introduce over 1 us of error (to accuracy!), but NTP will never show you this That doesn't apply to NTP to such an extent as PTP because it timestamps end of the reception (HW RX timestamps are transposed to the end of the packet), so the asymmetries in transmission lengths due to different link speeds should cancel out (unless the switches are cutting through in the faster->slower link direction, but that seems to be rare).
- Bender 1y agoThey state there is a problem but then state they are happy with what Chrony is doing so what exactly is the problem they are trying to solve? What on their network is requiring better than 200ns? or even 400ns for that matter? Not in theory but in reality? Also there are optimizations they are missing in this document [1] such as disabling EEE. On a more taboo note, while RasPi's can be great little time servers they have more drift and will have higher jitter but that should not matter for a home setup and should not be surprising. If jitter is their concern then they should consider using mini-pc's, disable cpuspeed, all power management and confine/lock the min/max speed to half the CPU capabilities and disable all services other than chrony. It will use more power but would address their concerns. They could also try different models of layer 2 switches. Consumer switches will add some artificial jitter and that varies wildly from make, model and even batch but again for a home network that should not matter. I think they are nitpicking. Perfect is the enemy of good, especially in a day and age when people prefer power saving over accuracy. [Edit] As a side note the aggressive min/max poll settings they are using can amplify the inefficiencies of consumer switches and NICs regardless of filter settings and that can make the graphs more chaotic. They should consider re-testing that on data-center class servers, server NICs and enterprise class switches or just reduce the polling to something reasonable for a home network minpoll 1 maxpoll 2 for client, minpoll 5 maxpoll 7 for edge talking to a dozen stratum 1's with a high combinelimit. Presend should not be required even with default ARP neighbor GC times and intervals. Oh and if you want to try something fun with the graphs, run chronyc makestep ever minute in cron on every node. yeah yeah I know why one would not do that and its just cheating. [1] - https://chrony-project.org/examples.html https://chrony-project.org/examples.html
- wowczarek 1y agoIn addition, for the purposes of characterising the system using NTP, ideally one should also either eavoid any ensembling / combining of sources because that's just pulling in multiple sources of noise, or it should be proven that doing so does not affect the final results, or if it does, then by how much. There's so much more that can be picked apart here because it's an absolute rabbit hole of a topic - for example, saturate the links a little or a little more, especially with bursty traffic in both directions (or do an 80-20 cycle), and watch those measurements go out the window and only with PTP-capable switches at every hop will you survive this. The Telecom industry has done it ad nauseam and for years with appropriate standardised measurements, test masks and requirements. And this whole business is also not fundamentally PTP vs. NTP because the principles are exactly the same, it's the fact that PTP was designed with hardware timestamping in mind and it would serve no purpose more useful than NTP had NTP gained support for one-step operation, hardware timestamping - and network assistance. But the default PTP profile uses known multicast groups and thus known destination MACs and it was the easiest entry into hardware packet matching - early "PTP-enabled" NICs only timestamped PTP packets (and most only multicast), only more modern ones allowed to timestamp all packets and that includes NTP. And as far as RasPi goes - for time sync, at least in terms of COTS equipment, Intel is king, but that's because they had smart people working hard for years to purposefully integrate time-aware functionalities into the architectures (Hey Kevin and team!) - invariant TSC, ART, culminating with PCIE PTM. But this is where aiming for the tens to single digit ns region. You can easily deliver sub-10 ns sync to a NIC, but a huge source of uncertainty is time transfer from your hardware-timestamping NIC to the OS clock. PTM is the only way to do this in hardware, otherwise, with Solarflare being the only NON-PTM exception I've worked with, comparing NIC to OS time is literally reading the time register on the NIC and the kernel time in quick succession in batches (granted, with local interrupts disabled), and then picking the pair of reads that seems to have taken the least amount of time. Unknowns on top of unknowns.
- myrandomcomment 1y agoThe paths on the network with the MLAG is NOT likely the issue. There is a serialization delay different between the NICs in the NTP servers and switches. 100M takes more time then 1G which takes more time then 10G which takes more time then 40G. Also the UI kit is store and forward (not sure on the 10G but the desktop one is) switching and the Arista kit is cut thought (at around ~500 byte packets IIRC). The end to end paths are not the same given the link speed difference and that is the source of the variations. The MLAG hashing is in hardware and will not have an effect, also IIRC you can set the LAG hash to be SRC/DST on both L2 and L3 even on a L2 link.