9 ms·

Clock synchronization is a nightmare

232 points by grep_it 9mo ago

koudelka 9mo ago
the Huygens algorithm is also worth a look https://www.usenix.org/system/files/conference/nsdi18/nsdi18-geng.pdf https://www.usenix.org/system/files/conference/nsdi18/nsdi18...
- danzheng 9mo ago
  A very clever part of the HUYGENS algorithm is that it doesn’t just sync clocks pair-wise, it leverages a natural network effect where a group of pair-wise synchronized clocks becomes transitively synchronized, helping reduce errors further without requiring specialized hardware. That’s one of the key reasons it can achieve ~100 nanoseconds of software-based sync on commodity networks. The authors’ work forms the basis of what the team at Clockwork.io is building, enabling accurate one-way delay measurements (rather than just RTT/2) that improve latency visibility and telemetry across CPU and GPU infrastructure
jeffbee 9mo ago
PTP requires support not only on your network, but also on your peripheral bus and inside your CPU. It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states.
- pclmulqdq 9mo ago
  You can if you just run PTP (almost) entirely on your NIC. The best PTP implementations take their packet timestamps at the MAC on the NIC and keep time based on that. Nothing about CPU processing is time-critical in that case.
  - jeffbee 9mo ago
    Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. The control loop for host PTP benefits from regularity. Anyway NICs that support PTP hardware timestamping may also use PCI LTR (latency tolerance reporting) to instruct the host operating system to disable high-exit-latency sleep features, and popular operating systems respect that.
    - DannyBee 9mo ago
      "Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. " I'm sorry, this is just moving the goalposts. You said "It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states." This is flat wrong, as pointed out. Now you are pedantically arguing that some NIC's that do PTP hardware timestamping might also use a feature that some operating systems might respect. That's a very far cry from "It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states". In most cases, people would just say "hey i was wrong about that but there are cases that i think matter where it falls down".
      - jeffbee 9mo ago
        I see nothing in your pair of unnecessarily belligerent comments that actually contradicts what I said. There are host-side features that enable the clock discipline you are observing, even if you are apparently not aware of them.
        DannyBee 9mo ago
        This is a really helpful contribution - if only everyone could be as smart as you. If mine are somehow too beligerent for you, which is hilarious given how arrogant and beligerent your initial comment and responses come off as (maybe you are not aware?), then perhaps you'd like to actually engage any of the other comments that point out how wrong you are in a meaningful way? Or are those too beligerent as well? Because you didn't respond to any of those, either.
        deleted 9mo ago
        [deleted]
    - Dylan16807 9mo ago
      > The control loop for host PTP benefits from regularity. How much regularity? If you sent PTP packets with 5 milliseconds of randomness in the scheduling, does that cause real problems? It's still going to have an accurate timestamp. > instruct the host operating system to disable high-exit-latency sleep features Why, though? You didn't explain this. As long as the packet got timestamped when it arrived, the CPU can ask the NIC how many nanoseconds ago that was, and correct for how long it was asleep. Right?
- rcxdude 9mo ago
  How so? If the NIC is processing the timestamps as it arrives/leaves on the wire, the latency and jitter in the rest of the system shouldn't matter.
- DannyBee 9mo ago
  PTP does not require support on your network beyond standard ethernet packet forwarding when used in ethernet mode. In multicast IP mode, with multiple switches, it requires what anything running multicast between switches/etc would require (IE some form of IGMP snopping or multicast routing or .....) In unicast IP mode, it requires nothing from your network. Therefore, i have no idea what it means to "require support on the network". I have used both ethernet and multicast PTP across a complete mishmash of brands and types and medias of switches, computers, etc, with no issues. The only thing that "support" might improve is more accurate path delay data through transparent clocks. If both master and slave do accurate hardware timestamping already, and the path between them is constant, it is easily possible to get +-50 nanoseconds without any transparent clock support. Here is the stats from a random embedded device running PTP i just accessed a second ago: Reference ID : 50545030 (PTP0) Stratum : 1 Ref time (UTC) : Sun Dec 28 02:47:25 2025 System time : 0.000000029 seconds slow of NTP time Last offset : -0.000000042 seconds RMS offset : 0.000000034 seconds Frequency : 8.110 ppm slow Residual freq : -0.000 ppm Skew : 0.003 ppm So this embedded ARM device, which is not special in any way, is maintaining time +-35ns of the grandmaster, and currently 30ns of GPS time. The card does not have an embedded hardware PTP clock, but it does do hardware timestamp and filtering. This grandmaster is an RPI with an intel chipset on it and the PPS input pin being used to discipline the chipset's clock. It stays within +-2ns (usually +-1ns) of GPS time. Obviously, holdover sucks, but not the point :) This qualifies as better-than-NTP for sure, and this setup has no network support. No transparent clocks, etc. These machines have multiple media transitions involved (fiber->ethernet), etc. The main thing transparent clock support provides in practice is dealing with highly variable delay. Either from mode of transport, number of packet processors in between your nodes, etc. Something that causes the delay to be hard to account for. The ethernet packet processing in ethernet mode is being handled in hardware by the switches and basically all network cards. IP variants would probably be hardware assisted but not fully offloaded on all cards, and just ignored on switches (assuming they are not really routers in disguise). The hardware timestamping is being done in the card (and the vast majority of ethernet cards have supported PTP harware timestamping for >1 decade at this point), and works perfectly fine with deep CPU sleep states. Some don't do hardware filtering so they essentially are processing more packets that necessary but .....
emptybits 9mo ago
Normally I would nod at the title. Having lived it. But I just watched/listened to a Richard Feynmann talk on the nature of time and clocks and the futility of "synchronizing" clocks. So I'm chuckling a bit. In the general sense, I mean. Yes yes, for practical purposes in the same reference frame on earth, it's difficult but there's hope. Now, in general ... synchronizing two clocks is ... meaningless? https://www.youtube.com/watch?v=zUHtlXA1f-w https://www.youtube.com/watch?v=zUHtlXA1f-w
- hinkley 9mo ago
  Einstein was worried about whether people in two different relativistic frames would see cause and effect reversed.
  - emptybits 9mo ago
    Wild. My layperson mind goes to a simple example, which may or may not be possible, but please tell me if this is the gist: Alice and Bob, in different reference frames, both witness events C and D occurring. Alice says C happened before D. Bob says D happened before C. They're both correct. (And good luck synchronizing your watches, Alice and Bob!)
    - mrkeen 9mo ago
      That will be the case when Alice stands close to where C happens, and Bob stands close to where D happens. It's a little trickier to imagine introducing cause-and-effect though. (Alice sees that C caused D to happen, Bob sees that D caused C to happen). I think a "light cone" is the thought-experiment to look up here.
      - hinkley 9mo ago
        If Bob and Alice are moving at half the speed of light in opposite directions.
      - ianburrell 9mo ago
        There is distinction between seeing when events happened, and when they really happened. The latter can be reconstructed by an observer. In special relativity, time is relative and when things actually happened can be different in different frames. Casually linked events are always really in the same order. But disconnected events can be seen in different orders depending on speed of observer.
yapyap 9mo ago
Love learning new things. This also explains why my casio clock sync starts skewing over time
maximinus_thrax 9mo ago
I wouldn't say it's a 'nightmare'. It's just more complicated than what regular folk think computers work when it comes to time sync. There's nothing nightmareish or scary about this, it's just using the best solution for your scenario, understanding limitations and adjusting expectations/requirements accordingly, perhaps relaxing consistency requirements. I worked on the NTP infra for a very large organization some time ago and the starriest thing I found was just how bad some of the clocks were on 'commodity hardware' but this just added a new parameter for triaging hardware for manufacturer replacement. This is an ok article but it's just so very superficial. It goes too wide for such a deep subject matter.
- blibble 9mo ago
  PTP isn't even that much more difficult, as long as you planned for it form the start you buy the hardware, plug it all in, and it works
  - geerlingguy 9mo ago
    Sometimes hardware that has PTP support in the specs doesn't perform very well though, so if you do things at scale, being able to validate things like switches and network card drivers is useful too! It's to the point timing server vendors I've spoken to have their own test labs where they have to validate network gear and then publish lists of recommended and tested configurations. Even some older cards where you'd think the PTP issues would be solved still have weird driver quirks in Linux!
  - varjag 9mo ago
    We once spent two weeks identifying a PTP handling bug in a particular Cisco switch firmware on a production site.
- hinkley 9mo ago
  I took to distributed systems like a duck to water. It was only much later that I figured out that while there are things I can figure out in one minute that took other people five, there were a lot of others that you will have to walk them through step by step or they would never get there. That really explained some interactions I’d had when I was younger. In particular I don’t think the intuitions necessary to do distributed computing well would come to someone who snoozed through physics, who never took intro to computer engineering.
mgaunard 9mo ago
Another protocol that's not mentioned is PPS and its variants, such as WhiteRabbit. A regular pulse is emitted from a specialized high-precision device, possibly over a specialized high-precision network. Enables picosecond accuracy (or at least sub-nano).
- nuccy 9mo ago
  As a user of WhiteRabbit, I can confirm a sub-10ps sync (two clocks phase lock) over 50km fiber connection for variable temperature of fiber (biggest problem of clock sync over fibers is temperature induced length change of the fiber itself, which needs to be measured and compensated).
  - RossBencina 9mo ago
    Out of interest, how do you measure a sub-10ps phase lock between devices 50km apart?
    - willis936 9mo ago
      The standards-compliant endpoints do all of the work. They count clock cycles for ping pong messages and share with each other the length of time so time-of-flight is tracked and compensated for.
      - deleted 9mo ago
        [deleted]
    - namibj 9mo ago
      Run 2 or 3 separate concurrent sync's and statistically compare the resulting clocks, for example.
    - nuccy 9mo ago
      As of now, for testing, the two WR endpoints are sitting on the same desk with 50km fiber in a thermal chamber (simulating temperature changes in the soil), but in future they will be separated indeed.
kobieps 9mo ago
Even just a single accurate clock is a nightmare... https://www.npr.org/2025/12/21/nx-s1-5651317/colorado-us-official-time-microseconds-nist-clocks https://www.npr.org/2025/12/21/nx-s1-5651317/colorado-us-off...
- Dylan16807 9mo ago
  I would not call "loses track of time if it's [partially] unplugged" a nightmare.
  - kobieps 9mo ago
    Haha fair, but in this case it was "loses the time if the power supply is interrupted"
    - Dylan16807 9mo ago
      I'm counting power loss as a type of being unplugged for that loose statement, even if technically they're different.
georgelyon 9mo ago
Unfortunate that the author doesn’t bring up FoundationDB version stamps, which to me feel like the right solution to the problem. Essentially, you can write a value you can’t read until after the transaction is committed and the synchronization infrastructure guarantees that value ends up being monotonically increasing per transaction. They use similar “write only” operations for atomic operations like increment.
- lll-o-lll 9mo ago
  Yes. A consistent total ordering is what you need (want) in distributed computing. Ultimately, causality is what is important, but consistent ordering of concurrent operations makes things much easier to work with.
  - josephg 9mo ago
    Consistent ordering of concurrent operations is easy though. Just detect this case (via logical clocks) then order using node ids or transaction ids if the logical clocks show the transactions as being concurrent. Am I missing something? This feels like a very solved problem. (I’ve worked on CRDTs where we have the same problem. There exist incredibly fast algorithms for this.)
    - lll-o-lll 9mo ago
      > Am I missing something? I don’t think so, I think it is solved in the general sense. However what Spanner does is unique, and it does use synchronised clocks in order to do it. However, Spanner does not solve the inter-continental acid database with high write throughput. So I don’t see it as ground breaking. CRDT’s are interesting, I’ve followed your work for a long time, but too constrained to solve this general problem I think.
    - georgelyon 9mo ago
      Yes, though the API of having a write-only value that is a monotonically increasing counter is much simpler than having to think about causality or logical clocks.
hinkley 9mo ago
Vector clocks are one of the other things Barbara Liskov is known for.
hinkley 9mo ago
> Google faced the clock synchronization problem at an unprecedented scale with Spanner, its globally distributed database. They needed strong consistency guarantees across data centers spanning continents, which requires knowing the order of transactions. > Here’s a video of me explaining this. Do you need a video? Do we need a 42 minute video to explain this? I generally agree with Feynman on this stuff. We let explanations be far more complex than they need to be for most things, and it makes the hunt for accidental complexity harder because everything looks almost as complex as the problems that need more study to divine what is actually going on there. For Spanner to be useful they needed a high transaction rate and in a distributed system that requires very tight grace periods for First Writer Wins. Tighter than you can achieve with NTP or system clocks. That’s it. That’s why they invented a new clock. Google puts it this way: Under external consistency, the system behaves as if all transactions run sequentially, even though Spanner actually runs them across multiple servers (and possibly in multiple datacenters) for higher performance and availability. But that’s a bit thick for people who don’t spend weeks or years thinking about distributed systems.
j_seigh 9mo ago
Ok,so people use NTP to "synchronize" their clocks and then write applications that assume the clocks are in exact sync and can use timestamps for synchronization, even though NTP can see the clocks aren't always in sync. Do I have that right?
- kccqzy 9mo ago
  If you are an engineer at Google dealing with Spanner, then you can in fact assume clocks are well synchronized and can use timestamps for synchronization. If you get commit timestamps from Spanner you can compare them to determine exactly which commit happened first. That’s a stronger guarantee than the typical Serializable database like postgresql: https://www.postgresql.org/docs/current/transaction-iso.html#XACT-SERIALIZABLE https://www.postgresql.org/docs/current/transaction-iso.html... That’s the radical developer simplicity promised by TrueTime mentioned in the article.
  - awesome_dude 9mo ago
    Isn't that because Google has its own atomic clocks, rather than NTP which is (generally) using publicly available atomic clocks?
    - deleted 9mo ago
      [deleted]
    - vlovich123 9mo ago
      More that they use GPS to synchronize the clocks. Having your own atomic clock doesn’t really improve your accuracy except for within the single data center you have it deployed (although I’m sure there’s techniques for synchronizing with low bounds against nearby atomic clocks + GPS to get really tight bound so they don’t need one in every data center)
  - idorosen 9mo ago
    Alternatively, you could guarantee the same synchronization using PPS and PTP to each host's DCD pin of their serial port or to specialized hardware such as modern PTP-enabled smart NICs/FPGAs that can accept PPS input. GPS+PPS gets you to within 20-80ns global synchronization depending on implementation (assuming you're all mostly in the same inertial frame), and allows you to make much stronger guarantees than TrueTime (due to higher precision distributed ordering guarantees, which translate to lower latency and higher throughput distributed writes).
forrestthewoods 9mo ago
Timesync isn’t a nightmare at all. But it is a deep rabbit hole. The best approach, imho, is to abandon the concept of a global time. All timestamps are wrt a specific clock. That clock will skew at a rate that varies with time. You can, hopefully, rely on any particular clock being monotonous! My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any clock to any clock. This is a lossy conversion that has jitter and can change with time. The fewer stops the better. I kinda don’t like PTP. Too complicated and requires specialized hardware. This article only touches on one class of timesync. An entirely separate class is timesync within a device. Your phone is a highly distributed compute system with many chips each of which has their own independent clock source. It’s a pain in the ass. You also have local timesync across devices such as wearables or robotics. Connecting to a PTP system with GPS and atomic clocks is not ideal (or necessary). TicSync is cool and useful. https://sci-hub.se/10.1109/icra.2011.5980112 https://sci-hub.se/10.1109/icra.2011.5980112
- DannyBee 9mo ago
  "I kinda don’t like PTP. Too complicated and requires specialized hardware." ????? I run PTP on everything from RPI's to you name it, over fiber, ethernet, etc. The main thing hardware gives is filtration of PTP packets or hardware timestamping. Neither is actually required, though some software has decided to require it. Additionally, something like 99% of sold gigabit or better chipsets since 2012 support it (I210 et al)
  - forrestthewoods 9mo ago
    Robots and VR headsets and wearables and microcontrollers and sensors and trackers and Linux and Windows oh my!
- RossBencina 9mo ago
  > I kinda don’t like PTP. Too complicated and requires specialized hardware. In my view the specialised hardware is just a way to get more accurate transmission and arrival timestamps. That's useful whether or not you use PTP. > My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any clock to any clock. This is a lossy conversion that has jitter and can change with time. This sounds like the "peer to peer" equivalent to PTP. It would require every node to maintain state about it's estimate (skew, slew, variance) of every other clock. I like the concept, but obviously it adds complexity to end-stations beyond what PTP requires (i.e. increases the hardware cost of embedded implementations). Such a system would also need to model the network topology, or control routing (as PTP does), because packets traversing different routes to the same host will experience different delay and jitter statistics. > TicSync is cool I hadn't seen this before, but I have implemented similar convex-hull based methods for clock recovery. I agree this is obviously a good approach. Thanks for sharing.
a_t48 9mo ago
Clock sync is such a nightmare in robotics. Most OSes happily will skew/jump to get the time correct. Time jumps (especially backwards) will crash most robotics stacks. You might decide to ensure that you have synced time before starting the stack. Great, now your timestamps are mostly accurate, except what happens when you've used GPS as your time source, and you start indoors? Robot hangs forever. Hot take: I've seen this and enough other badly configured time sync settings that I want to ban system time from robotics systems - time from startup only! If you want to know what the real world time was for a piece of data after, write what your epoch is once you have a time sync, and add epoch+start time.
- awesome_dude 9mo ago
  THIS is what will save us from the robot uprising!
- michaelt 9mo ago
  If your requirements are “must have accurate time, must start with an inaccurate time, must not step time during operation, no atomic clocks, must not require a network connection, or a WWVB signal, must work without a GPS signal” then yes, you need to relax your requirements. But it doesn’t have to be the first requirement you relax.
  - a_t48 9mo ago
    If it has a GPS already, it’s really easy to fall into the trap of just using it, but point taken. Then main requirement is accurate moment to moment time. Using GPS as the master clock mostly makes sense there.
- RossBencina 9mo ago
  C++11 distinguishes system_clock from steady_clock. As you say, using system_clock is a bug.
Asmod4n 9mo ago
Does wall clock time matter for anything but logging? For everything else one could just create any form „time“ to keep stuff in sync, no?
- tbrownaw 9mo ago
  Isn't it also useful for checking validity periods for stuff like TLS certs or JWTs or Kerberos tickets?
  - Asmod4n 9mo ago
    We could use some made up „time“ for that since it’s not made for human consumption just sync between different systems.
    - tbrownaw 9mo ago
      I suppose this is changing with TLS certs moving towards ephemerality, but they used to have an entry on someone's calendar for renewal.
NelsonMinar 9mo ago
On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at how in my lifetime I from "no clock is ever set right" to assuming most anything was within a second of true time.
- jasonwatkinspdx 9mo ago
  At this point the only clock in my life that doesn't auto set is the one on my stove, and that's because I abhor internet connected kitchen appliances.
  - amelius 9mo ago
    Same here. I wish there was an easy way around it (that doesn't require me to play sysadmin in my spare time).
    - jaggederest 9mo ago
      In the 80s my uncle had digital clocks that used an antenna to tune into the atomic clock time signal that (was/is?) broadcast nationwide. I've long wished that it was incorporated into stoves, microwaves, essentially everything that isn't an internet device (yet... sigh) Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of a pin these days.
      - jasonwatkinspdx 9mo ago
        > Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of a pin these days. Unfortunately there's no real way to cheat physics as far as shrinking a wavelength goes. With RF antennas about the best you can do is a major dimension 1/10th the frequency of interest.
shomp 9mo ago
Absolute synchronization impossible?? Challenge accepted.
- nuccy 9mo ago
  Nature (laws of physics) is agains you on this: it is in fact impossible for everyone. What is in sync for some observers can be out of sync for others (depends on where they are, i.e. gravity, and how they relatively move). See general and special relativity principle of simultaneity [1]. 1. https://en.wikipedia.org/wiki/Relativity_of_simultaneity https://en.wikipedia.org/wiki/Relativity_of_simultaneity
  - Enginerrrd 9mo ago
    I think you just nerd-sniped me but I’m not convinced it’s impossible to assign a consistent ordering to events with relativistic separations. For starters, the spacetime interval between two events IS a Lorentz invariant quantity. That could probably be used to establish a universal order for timelike separations between events. I suspect that you could use a reference clock, like a pulsar or something to act as an event against which to measure the spacetime interval to other events, and use that for ordering. Any events separated by a light-like interval are essentially simultaneous to all observers under that measure. The problem comes for events with a space like or light like separation. In that case, the spacetime interval is still conserved, but I’m not sure how you assign order to them. Perhaps the same system works without modification, but I’m not sure.
    - tobias2014 9mo ago
      For any space-like event you can find reference frames where things happen in different order. For the time-like situation you described the order indeed exists within the cone, which is to say that causality exists.
      - Enginerrrd 9mo ago
        You can still order them with the spacetime interval compared to a reference event, even for space like separated events. It allows for differing elements of the set to share the same value but so does using time alone. It just also allows every observer to agree on the ordering. Bc Assigning a distance function to elements of a set is a common way to do that in fact. It doesn’t work with just a time coordinate or space coordinate, because that’s effectively a Euclidean metric. You just have to contend with a few nonintuitive aspects but it’s not so bad.
josephg 9mo ago
> When two transactions happen at nearly the same time on different nodes, the database must determine which happened first. If clocks are out of sync, the database might order them incorrectly, violating consistency guarantees. This is only true if you use wall clock time as part of your database’s consistency algorithm. Generally I think this is a huge mistake. It’s almost always much easier to swap to a logical clock - which doesn’t care about wall time. And then you don’t have to worry about ntp. The basic idea is this: event A happened before event B iff A (or something that happened after A) was observed by the node that generated B before B was generated. As a result, you end up with a dag of events - kind of like git. Some events aren’t ordered relative to one another. (We say, they happened concurrently). If you ever need a global order for all events, you can deterministically pick an arbitrary order for concurrent events by comparing ids or something. And this will give you a total order that will be the same on all peers. If you make database events work like this, time is a little more complex. (It’s a graph traversal rather than simple numbers). But as a result the system clock doesn’t matter. No need to worry about atomic clocks, skew, drift, monotonicity, and all of that junk. It massively simplifies your system design.
- johnisgood 9mo ago
  Related in many ways: https://www.erlang.org/docs/22/apps/erts/time_correction https://www.erlang.org/docs/22/apps/erts/time_correction Also I still remember having fun with the "Determine the order of events by saving a tuple containing monotonic time and a strictly monotonically increasing integer as follows" part.
- b112 9mo ago
  Unfortunately, some of us have to deal with things like billing, transaction timing to validate what a client's logs might have on their systems, and so on. My take on this is that second timing is close enough for this. And that all my internal systems need agree on the time. So if I'm off by 200ms or some blather from the rest of the world, I'm not overly concerned. I am concerned, however, if a random internal system is not synced to my own ntp servers. This doesn't mean I don't keep our servers synced, just that being off by some manner of ms doesn't bother me inordinately. And when it comes to timing of events, yes, auto-increment IDs or some such are easier to deal with.
dmazin 9mo ago
I highly recommend anyone to look up how PTP works and how it compares to NTP. Clock sync is very interesting. When I joined an HFT company, first thing I did was understand this stuff. We care about it a lot[1]. If you want a specific question to answer, answer this: why does PTP need hardware timestamping to achieve high precision (where the network card itself assigns timestamps to packets, rather than having the kernel do it as part of TCP/IP processing)? If we use software timestamps, why can we do microsecond precision at best? If you understand this, it goes a very long way to understanding the core ideas behind precise clock sync. Once you have a solid understanding of PTP, look into White Rabbit. They’re able to sync two clocks with sub-ns precision. In case that isn’t obvious, that is absolutely insane. [1] So do a lot of people. For example audio engineers. Once, an audio engineer absolutely talked my ear off about ptp. I had no idea that audio people understood clock sync so well but they do!
- RossBencina 9mo ago
  > So do a lot of people. For example audio engineers. Indeed. PTP (various, not-necessarily compatible, versions) is at the core of modern ethernet-based audio networking: Dante (proprietary, PTP: IEEE 1588 v1), AVB (IEEE standard, PTP: 802.1AS), AES67 (AES standard, PTP: IEEE 1588 v2). And now the scope of the AVB protocol stack has been expanded to TSN for industrial and automotive time sensitive network applications.
  - dmazin 9mo ago
    Yeah, the audio engineer then talked my ear off about networking!
- baby_souffle 9mo ago
  Not just audio, anybody in the live events / production space needs all equipment marching in lock step.
  - 8n4vidtmkvmk 9mo ago
    If it's for an event, can they not bring all the devices together in close proximity and sync them somehow? That at least removes network delays
didgetmaster 9mo ago
Reminds me of the old saying: 'If you have just one watch/clock, then you always know what time it is; but if you have two of them, then you are never sure!'
pdeva1 9mo ago
AWS has the Google TrueTime equivalent precision clock available for public use[1] which makes this problem much easier to solve now. Auora DSQL uses it. Even third party db's like YugabyteDb make use of it. [1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time...
gdcohen 9mo ago
Take a look at Exploiting a Natural Network Effect for Scalable, Fine-grained Clock Synchronization - https://www.usenix.org/conference/nsdi18/presentation/geng https://www.usenix.org/conference/nsdi18/presentation/geng (commercial version by the same authors - clockwork.io).
sreekanth850 9mo ago
In physics, time is local and relative, independent events don’t need a global ordering. Distributed databases shouldn’t require one either. The idea of a single global time comes from 1980s single-node database semantics, where serializability implied one universal execution order. When that model was lifted into distributed systems, researchers introduced global clocks and timestamp coordination to preserve those guarantees, not because distributed systems fundamentally need it. It’s time we rethink this., Only operations that touch the same piece of data require ordering. Everything else should follow causality like the physical universe, independent events don’t need to agree on sequence, only dependent ones do. Global clocks exist because some databases forced serializable cross-object transactions onto distributed systems, not because nature requires it. Edit: I welcome for a discussion with people who disagree and downvote.
- otterley 9mo ago
  You can’t be certain that any given mutating operation you perform now won’t be relied upon for some future operation, unless the two operations are performed in entirely different domains of data. Even “not touching (by which I assume you mean mutating) the same data” isn’t enough. If I update A in thread 0 from 1 to 2, then I update B in thread 1 to the value of A+1, then the value of B could end up being 2 or 3, depending on whether the update of A reached thread 1.
  - sreekanth850 9mo ago
    In distributed systems, dependencies flow forward, not backward. Causal dependency only exists when an operation actually references earlier state. If B = A+1, then yes, B is causally dependent on A and they must share an order. But that dependency is created by the application logic, not assumed globally in advance. We shouldn’t impose a universal timeline just because some future operation might depend on some past one. Dependencies should be explicit and local: if two operations interact, they share a causal scope; if they don’t, they shouldn’t pay the cost of coordination.
    - otterley 9mo ago
b112 9mo ago
The article doesn't cover the inane stupid that is: * NTP pool server usage requires using DNS * people have DNSSEC setup, which requires accurate time or it fails So if your clock is off, you cannot lookup NTP pool servers via DNS, and therefore cannot set your clock. This sheer stupid has been discussed with package maintainers of major distros, with ntpsec, and the result is a mere shrug. Often, the answer is "but doesn't your device have a battery backed clock?", which is quite unhelpful. Many devices (routers, IOT devices, small boards, or older machines, etc) don't have a battery backed clock, or alternatively the battery may just have died. Beyond that, the ntpsec codebase has a horrible bug where if DNS is not available when ntpsec starts, pool server addresses are never, ever retried. So if you have a complete power-fail in a datacentre rack, and your firewalls take a little longer to boot than your machines, you'll have to manually restart ntpsec to even get it to ever sync. When discussing this bug the ntpsec lads were confused that DNS might not exist at times. Long story short, make sure you aren't using DNS in any capacity, in NTP configs, and most especially in ntpsec configs. One good source is just using the IPs provided by NIST. Pool servers may seem fine, but I'd trust IPs assigned to NIST to exist longer than any DNS anyhow. EG, for decades.
- ectospheno 9mo ago
  I gave up and just got a gps ntp server. If you have a wood house then you can even get away with an inside antenna.
simonebrunozzi 9mo ago
Back when I was studying computer science, I was taking the OS exam and the part about Lamport timestamp [0] was optional, but I had studied it because I loved it. When I mentioned it to my professor, he was so happy to hear something new that day that he asked me to describe it in details. This was the year 2001. Many years later, in 2020, I ended up living in San Francisco, and I had the fortune to meet Leslie Lamport after I sent him a cold email. Lovely and smart guy. This is the text of the first part of that email, just for your curiosity: Hey Leslie! You have accompanied me for more than 20 years. I first met your name when studying Lamport timestamps. And then on, and on, and on, up to a few minutes ago, when I realized that you are also behind the paper and the title of "Byzantine Generals problem", renamed after the "Albanian" generals to the suggestion of Jack Goldberg. Who is he? [1] [0]: https://en.wikipedia.org/wiki/Lamport_timestamp https://en.wikipedia.org/wiki/Lamport_timestamp [1]: Jack Goldberg (now retired) was a computer scientist and Lamport's manager at SRI.
eatsome 9mo ago
For an article written about time, I would have thought there'd be a timestamp on the blog post. Just something to think about if someone stumbles upon this in a few years.
amiune 9mo ago
Related: https://632nm.com/episodes/why-syncing-atomic-clocks-is-virtually-impossible-judah-levine-on-utc https://632nm.com/episodes/why-syncing-atomic-clocks-is-virt... As a teacher I love the way Judah Levine explains
user3939382 9mo ago
That’s because neither discrete time nor synchronous network comms exist.
layer8 9mo ago
> The good news is that the International Bureau of Weights and Measures has decided to stop adding leap seconds by 2035. This is not entirely correct. What has been agreed is to allow deviations of more than one second after 2035, so that clocks have to be adjusted less frequently (on the order of every 50-100 years is the intention). However, the allowable deviation, and how to adjust clocks when it is exceeded, has yet to be decided.
danzheng 9mo ago
One thing missing in the blogpost is in practice you see many large orgs, especially in finance, living with multiple time domains. For example, on-prem trading systems almost always use PTP or PPS for sub-microsecond timestamping, often on dedicated networks to reduce jitter (for meeting regulatory requirements like MiFID II and CAT) while the rest of their infra (in on-prem and cloud) just runs NTP for millisecond-class sync. Both protocols are fundamentally sensitive to network conditions — the mean offset may look fine, but outliers due to congestion/jitter can be very poor. The consequence of having multiple time domains is pretty painful when you need to reconcile logs or transaction histories across systems with different sync accuracy. Millisecond NTP logs and sub-microsecond PTP logs don’t line up cleanly, so correlating events end-to-end can become guesswork rather than deterministic ordering. If you want reliable cross-system telemetry and audit trails, you'll need a single, high-accuracy time sync approach across your whole stack.
sureshvoz 9mo ago
This is a great breakdown, and it’s worth noting that we are hitting a "microsecond wall" in modern GPU clusters that makes standard NTP effectively obsolete. In distributed training (LLMs), the bottleneck is no longer just disk I/O or CPU cycles—it’s the "straggler problem" during collective communication (like All-Reduce). When you’re running on 400Gbps+ RoCE (RDMA over Converged Ethernet) networks, the network "wire time" is often lower than the clock jitter on a standard Linux kernel. If your clocks are skewed by even 2-3 milliseconds, your telemetry becomes essentially useless. It looks like packets are arriving before they were sent, or worse, your profiling tools can’t accurately pinpoint which GPU is stalling the rest of the 16,384-node fleet. We’ve reached a point where microsecond-accurate clocks isn't just a requirement for HFT firms; it’s becoming the baseline for anyone trying to keep $100s of millions of NVidia GPUs from idling while they wait for a collective sync.
- perryizgr8 9mo ago
  If you have network infrastructure that supports 400G I'm pretty sure it has solid PTP built in. And as far as I remember from my networking days setting it up is almost as simple as setting up NTP, you just need a single machine with a GPS lock.
j_seigh 9mo ago
The comments about HFT needing tightly synchronized clocks got me thinking. Back in the day, way back in the 80's, IBM replaced the VM with VMXA. VM could trap and emulate all the important instructions since they were privileged instructions except one, the STCK (store clock) instruction. So virtual machines couldn't set their virtual clocks so they were always in sync. VMXA used new hw features that let you set the virtual clock. You could specify an offset to the system clock. But some of IBM's biggest customers depended on all the virtual machines clocks always being in sync. So VMXA had to add an option to disallow setting the clock for specified virtual machines. Except all of development knew how trivial it was to trap or modify the STCK's to produce a timestamp of you choosing. This was before it was common knowledge the client code should never be trusted. But nobody enlightened IBM corporate management. It was a serious career limiting move at IBM. It didn't matter if you were right. So I'm pretty sure some serious fortunes were made as a result. So the question for HFT is; are they using and trusting client timestamps, or are the timestamps being generated on the market maker's servers? If the latter, how would the customer know?
jmpman 9mo ago
Back in the early 2000s I was programming on an IBM AIX server. Multicore, maybe multiprocessor and within the same machine, the clocks were skewed between the processors. If you’d dispatch a process, and then check its outstanding running time, it would differ depending upon which processor you’d check from, and of course it was a signed type, and then we would get negative values, which sent our code down the wrong path.

232 points by grep_it 9mo ago

the Huygens algorithm is also worth a look https://www.usenix.org/system/files/conference/nsdi18/nsdi18-geng.pdf https://www.usenix.org/system/files/conference/nsdi18/nsdi18...

danzheng 9mo ago
A very clever part of the HUYGENS algorithm is that it doesn’t just sync clocks pair-wise, it leverages a natural network effect where a group of pair-wise synchronized clocks becomes transitively synchronized, helping reduce errors further without requiring specialized hardware. That’s one of the key reasons it can achieve ~100 nanoseconds of software-based sync on commodity networks. The authors’ work forms the basis of what the team at Clockwork.io is building, enabling accurate one-way delay measurements (rather than just RTT/2) that improve latency visibility and telemetry across CPU and GPU infrastructure

jeffbee 9mo ago

PTP requires support not only on your network, but also on your peripheral bus and inside your CPU. It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states.

pclmulqdq 9mo ago
You can if you just run PTP (almost) entirely on your NIC. The best PTP implementations take their packet timestamps at the MAC on the NIC and keep time based on that. Nothing about CPU processing is time-critical in that case.
- jeffbee 9mo ago
  Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. The control loop for host PTP benefits from regularity. Anyway NICs that support PTP hardware timestamping may also use PCI LTR (latency tolerance reporting) to instruct the host operating system to disable high-exit-latency sleep features, and popular operating systems respect that.
  - DannyBee 9mo ago
    "Well, if the goal is for software running on the host CPU to know the time accurately, then it does matter. " I'm sorry, this is just moving the goalposts. You said "It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states." This is flat wrong, as pointed out. Now you are pedantically arguing that some NIC's that do PTP hardware timestamping might also use a feature that some operating systems might respect. That's a very far cry from "It can't achieve better-than-NTP results without disabling PCI power saving features and deep CPU sleep states". In most cases, people would just say "hey i was wrong about that but there are cases that i think matter where it falls down".
    - jeffbee 9mo ago
      I see nothing in your pair of unnecessarily belligerent comments that actually contradicts what I said. There are host-side features that enable the clock discipline you are observing, even if you are apparently not aware of them.
      - DannyBee 9mo ago
        This is a really helpful contribution - if only everyone could be as smart as you. If mine are somehow too beligerent for you, which is hilarious given how arrogant and beligerent your initial comment and responses come off as (maybe you are not aware?), then perhaps you'd like to actually engage any of the other comments that point out how wrong you are in a meaningful way? Or are those too beligerent as well? Because you didn't respond to any of those, either.
        deleted 9mo ago
        [deleted]
  - Dylan16807 9mo ago
    > The control loop for host PTP benefits from regularity. How much regularity? If you sent PTP packets with 5 milliseconds of randomness in the scheduling, does that cause real problems? It's still going to have an accurate timestamp. > instruct the host operating system to disable high-exit-latency sleep features Why, though? You didn't explain this. As long as the packet got timestamped when it arrived, the CPU can ask the NIC how many nanoseconds ago that was, and correct for how long it was asleep. Right?
rcxdude 9mo ago
How so? If the NIC is processing the timestamps as it arrives/leaves on the wire, the latency and jitter in the rest of the system shouldn't matter.
DannyBee 9mo ago
PTP does not require support on your network beyond standard ethernet packet forwarding when used in ethernet mode. In multicast IP mode, with multiple switches, it requires what anything running multicast between switches/etc would require (IE some form of IGMP snopping or multicast routing or .....) In unicast IP mode, it requires nothing from your network. Therefore, i have no idea what it means to "require support on the network". I have used both ethernet and multicast PTP across a complete mishmash of brands and types and medias of switches, computers, etc, with no issues. The only thing that "support" might improve is more accurate path delay data through transparent clocks. If both master and slave do accurate hardware timestamping already, and the path between them is constant, it is easily possible to get +-50 nanoseconds without any transparent clock support. Here is the stats from a random embedded device running PTP i just accessed a second ago: Reference ID : 50545030 (PTP0) Stratum : 1 Ref time (UTC) : Sun Dec 28 02:47:25 2025 System time : 0.000000029 seconds slow of NTP time Last offset : -0.000000042 seconds RMS offset : 0.000000034 seconds Frequency : 8.110 ppm slow Residual freq : -0.000 ppm Skew : 0.003 ppm So this embedded ARM device, which is not special in any way, is maintaining time +-35ns of the grandmaster, and currently 30ns of GPS time. The card does not have an embedded hardware PTP clock, but it does do hardware timestamp and filtering. This grandmaster is an RPI with an intel chipset on it and the PPS input pin being used to discipline the chipset's clock. It stays within +-2ns (usually +-1ns) of GPS time. Obviously, holdover sucks, but not the point :) This qualifies as better-than-NTP for sure, and this setup has no network support. No transparent clocks, etc. These machines have multiple media transitions involved (fiber->ethernet), etc. The main thing transparent clock support provides in practice is dealing with highly variable delay. Either from mode of transport, number of packet processors in between your nodes, etc. Something that causes the delay to be hard to account for. The ethernet packet processing in ethernet mode is being handled in hardware by the switches and basically all network cards. IP variants would probably be hardware assisted but not fully offloaded on all cards, and just ignored on switches (assuming they are not really routers in disguise). The hardware timestamping is being done in the card (and the vast majority of ethernet cards have supported PTP harware timestamping for >1 decade at this point), and works perfectly fine with deep CPU sleep states. Some don't do hardware filtering so they essentially are processing more packets that necessary but .....

emptybits 9mo ago

Normally I would nod at the title. Having lived it. But I just watched/listened to a Richard Feynmann talk on the nature of time and clocks and the futility of "synchronizing" clocks. So I'm chuckling a bit. In the general sense, I mean. Yes yes, for practical purposes in the same reference frame on earth, it's difficult but there's hope. Now, in general ... synchronizing two clocks is ... meaningless? https://www.youtube.com/watch?v=zUHtlXA1f-w https://www.youtube.com/watch?v=zUHtlXA1f-w

hinkley 9mo ago
Einstein was worried about whether people in two different relativistic frames would see cause and effect reversed.
- emptybits 9mo ago
  Wild. My layperson mind goes to a simple example, which may or may not be possible, but please tell me if this is the gist: Alice and Bob, in different reference frames, both witness events C and D occurring. Alice says C happened before D. Bob says D happened before C. They're both correct. (And good luck synchronizing your watches, Alice and Bob!)
  - mrkeen 9mo ago
    That will be the case when Alice stands close to where C happens, and Bob stands close to where D happens. It's a little trickier to imagine introducing cause-and-effect though. (Alice sees that C caused D to happen, Bob sees that D caused C to happen). I think a "light cone" is the thought-experiment to look up here.
    - hinkley 9mo ago
      If Bob and Alice are moving at half the speed of light in opposite directions.
    - ianburrell 9mo ago
      There is distinction between seeing when events happened, and when they really happened. The latter can be reconstructed by an observer. In special relativity, time is relative and when things actually happened can be different in different frames. Casually linked events are always really in the same order. But disconnected events can be seen in different orders depending on speed of observer.

yapyap 9mo ago

Love learning new things. This also explains why my casio clock sync starts skewing over time

maximinus_thrax 9mo ago

I wouldn't say it's a 'nightmare'. It's just more complicated than what regular folk think computers work when it comes to time sync. There's nothing nightmareish or scary about this, it's just using the best solution for your scenario, understanding limitations and adjusting expectations/requirements accordingly, perhaps relaxing consistency requirements. I worked on the NTP infra for a very large organization some time ago and the starriest thing I found was just how bad some of the clocks were on 'commodity hardware' but this just added a new parameter for triaging hardware for manufacturer replacement. This is an ok article but it's just so very superficial. It goes too wide for such a deep subject matter.

blibble 9mo ago
PTP isn't even that much more difficult, as long as you planned for it form the start you buy the hardware, plug it all in, and it works
- geerlingguy 9mo ago
  Sometimes hardware that has PTP support in the specs doesn't perform very well though, so if you do things at scale, being able to validate things like switches and network card drivers is useful too! It's to the point timing server vendors I've spoken to have their own test labs where they have to validate network gear and then publish lists of recommended and tested configurations. Even some older cards where you'd think the PTP issues would be solved still have weird driver quirks in Linux!
- varjag 9mo ago
  We once spent two weeks identifying a PTP handling bug in a particular Cisco switch firmware on a production site.
hinkley 9mo ago
I took to distributed systems like a duck to water. It was only much later that I figured out that while there are things I can figure out in one minute that took other people five, there were a lot of others that you will have to walk them through step by step or they would never get there. That really explained some interactions I’d had when I was younger. In particular I don’t think the intuitions necessary to do distributed computing well would come to someone who snoozed through physics, who never took intro to computer engineering.

mgaunard 9mo ago

Another protocol that's not mentioned is PPS and its variants, such as WhiteRabbit. A regular pulse is emitted from a specialized high-precision device, possibly over a specialized high-precision network. Enables picosecond accuracy (or at least sub-nano).

nuccy 9mo ago
As a user of WhiteRabbit, I can confirm a sub-10ps sync (two clocks phase lock) over 50km fiber connection for variable temperature of fiber (biggest problem of clock sync over fibers is temperature induced length change of the fiber itself, which needs to be measured and compensated).
- RossBencina 9mo ago
  Out of interest, how do you measure a sub-10ps phase lock between devices 50km apart?
  - willis936 9mo ago
    The standards-compliant endpoints do all of the work. They count clock cycles for ping pong messages and share with each other the length of time so time-of-flight is tracked and compensated for.
    - deleted 9mo ago
      [deleted]
  - namibj 9mo ago
    Run 2 or 3 separate concurrent sync's and statistically compare the resulting clocks, for example.
  - nuccy 9mo ago
    As of now, for testing, the two WR endpoints are sitting on the same desk with 50km fiber in a thermal chamber (simulating temperature changes in the soil), but in future they will be separated indeed.

kobieps 9mo ago

Even just a single accurate clock is a nightmare... https://www.npr.org/2025/12/21/nx-s1-5651317/colorado-us-official-time-microseconds-nist-clocks https://www.npr.org/2025/12/21/nx-s1-5651317/colorado-us-off...

Dylan16807 9mo ago
I would not call "loses track of time if it's [partially] unplugged" a nightmare.
- kobieps 9mo ago
  Haha fair, but in this case it was "loses the time if the power supply is interrupted"
  - Dylan16807 9mo ago
    I'm counting power loss as a type of being unplugged for that loose statement, even if technically they're different.

georgelyon 9mo ago

Unfortunate that the author doesn’t bring up FoundationDB version stamps, which to me feel like the right solution to the problem. Essentially, you can write a value you can’t read until after the transaction is committed and the synchronization infrastructure guarantees that value ends up being monotonically increasing per transaction. They use similar “write only” operations for atomic operations like increment.

lll-o-lll 9mo ago
Yes. A consistent total ordering is what you need (want) in distributed computing. Ultimately, causality is what is important, but consistent ordering of concurrent operations makes things much easier to work with.
- josephg 9mo ago
  Consistent ordering of concurrent operations is easy though. Just detect this case (via logical clocks) then order using node ids or transaction ids if the logical clocks show the transactions as being concurrent. Am I missing something? This feels like a very solved problem. (I’ve worked on CRDTs where we have the same problem. There exist incredibly fast algorithms for this.)
  - lll-o-lll 9mo ago
    > Am I missing something? I don’t think so, I think it is solved in the general sense. However what Spanner does is unique, and it does use synchronised clocks in order to do it. However, Spanner does not solve the inter-continental acid database with high write throughput. So I don’t see it as ground breaking. CRDT’s are interesting, I’ve followed your work for a long time, but too constrained to solve this general problem I think.
  - georgelyon 9mo ago
    Yes, though the API of having a write-only value that is a monotonically increasing counter is much simpler than having to think about causality or logical clocks.

hinkley 9mo ago

Vector clocks are one of the other things Barbara Liskov is known for.

hinkley 9mo ago

> Google faced the clock synchronization problem at an unprecedented scale with Spanner, its globally distributed database. They needed strong consistency guarantees across data centers spanning continents, which requires knowing the order of transactions. > Here’s a video of me explaining this. Do you need a video? Do we need a 42 minute video to explain this? I generally agree with Feynman on this stuff. We let explanations be far more complex than they need to be for most things, and it makes the hunt for accidental complexity harder because everything looks almost as complex as the problems that need more study to divine what is actually going on there. For Spanner to be useful they needed a high transaction rate and in a distributed system that requires very tight grace periods for First Writer Wins. Tighter than you can achieve with NTP or system clocks. That’s it. That’s why they invented a new clock. Google puts it this way: Under external consistency, the system behaves as if all transactions run sequentially, even though Spanner actually runs them across multiple servers (and possibly in multiple datacenters) for higher performance and availability. But that’s a bit thick for people who don’t spend weeks or years thinking about distributed systems.

j_seigh 9mo ago

Ok,so people use NTP to "synchronize" their clocks and then write applications that assume the clocks are in exact sync and can use timestamps for synchronization, even though NTP can see the clocks aren't always in sync. Do I have that right?

kccqzy 9mo ago
If you are an engineer at Google dealing with Spanner, then you can in fact assume clocks are well synchronized and can use timestamps for synchronization. If you get commit timestamps from Spanner you can compare them to determine exactly which commit happened first. That’s a stronger guarantee than the typical Serializable database like postgresql: https://www.postgresql.org/docs/current/transaction-iso.html#XACT-SERIALIZABLE https://www.postgresql.org/docs/current/transaction-iso.html... That’s the radical developer simplicity promised by TrueTime mentioned in the article.
- awesome_dude 9mo ago
  Isn't that because Google has its own atomic clocks, rather than NTP which is (generally) using publicly available atomic clocks?
  - deleted 9mo ago
    [deleted]
  - vlovich123 9mo ago
    More that they use GPS to synchronize the clocks. Having your own atomic clock doesn’t really improve your accuracy except for within the single data center you have it deployed (although I’m sure there’s techniques for synchronizing with low bounds against nearby atomic clocks + GPS to get really tight bound so they don’t need one in every data center)
- idorosen 9mo ago
  Alternatively, you could guarantee the same synchronization using PPS and PTP to each host's DCD pin of their serial port or to specialized hardware such as modern PTP-enabled smart NICs/FPGAs that can accept PPS input. GPS+PPS gets you to within 20-80ns global synchronization depending on implementation (assuming you're all mostly in the same inertial frame), and allows you to make much stronger guarantees than TrueTime (due to higher precision distributed ordering guarantees, which translate to lower latency and higher throughput distributed writes).

forrestthewoods 9mo ago

Timesync isn’t a nightmare at all. But it is a deep rabbit hole. The best approach, imho, is to abandon the concept of a global time. All timestamps are wrt a specific clock. That clock will skew at a rate that varies with time. You can, hopefully, rely on any particular clock being monotonous! My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any clock to any clock. This is a lossy conversion that has jitter and can change with time. The fewer stops the better. I kinda don’t like PTP. Too complicated and requires specialized hardware. This article only touches on one class of timesync. An entirely separate class is timesync within a device. Your phone is a highly distributed compute system with many chips each of which has their own independent clock source. It’s a pain in the ass. You also have local timesync across devices such as wearables or robotics. Connecting to a PTP system with GPS and atomic clocks is not ideal (or necessary). TicSync is cool and useful. https://sci-hub.se/10.1109/icra.2011.5980112 https://sci-hub.se/10.1109/icra.2011.5980112

DannyBee 9mo ago
"I kinda don’t like PTP. Too complicated and requires specialized hardware." ????? I run PTP on everything from RPI's to you name it, over fiber, ethernet, etc. The main thing hardware gives is filtration of PTP packets or hardware timestamping. Neither is actually required, though some software has decided to require it. Additionally, something like 99% of sold gigabit or better chipsets since 2012 support it (I210 et al)
- forrestthewoods 9mo ago
  Robots and VR headsets and wearables and microcontrollers and sensors and trackers and Linux and Windows oh my!
RossBencina 9mo ago
> I kinda don’t like PTP. Too complicated and requires specialized hardware. In my view the specialised hardware is just a way to get more accurate transmission and arrival timestamps. That's useful whether or not you use PTP. > My mental model is that you form a connected graph of clocks and this allows you to convert arbitrary timestamps from any clock to any clock. This is a lossy conversion that has jitter and can change with time. This sounds like the "peer to peer" equivalent to PTP. It would require every node to maintain state about it's estimate (skew, slew, variance) of every other clock. I like the concept, but obviously it adds complexity to end-stations beyond what PTP requires (i.e. increases the hardware cost of embedded implementations). Such a system would also need to model the network topology, or control routing (as PTP does), because packets traversing different routes to the same host will experience different delay and jitter statistics. > TicSync is cool I hadn't seen this before, but I have implemented similar convex-hull based methods for clock recovery. I agree this is obviously a good approach. Thanks for sharing.

a_t48 9mo ago

Clock sync is such a nightmare in robotics. Most OSes happily will skew/jump to get the time correct. Time jumps (especially backwards) will crash most robotics stacks. You might decide to ensure that you have synced time before starting the stack. Great, now your timestamps are mostly accurate, except what happens when you've used GPS as your time source, and you start indoors? Robot hangs forever. Hot take: I've seen this and enough other badly configured time sync settings that I want to ban system time from robotics systems - time from startup only! If you want to know what the real world time was for a piece of data after, write what your epoch is once you have a time sync, and add epoch+start time.

awesome_dude 9mo ago
THIS is what will save us from the robot uprising!
michaelt 9mo ago
If your requirements are “must have accurate time, must start with an inaccurate time, must not step time during operation, no atomic clocks, must not require a network connection, or a WWVB signal, must work without a GPS signal” then yes, you need to relax your requirements. But it doesn’t have to be the first requirement you relax.
- a_t48 9mo ago
  If it has a GPS already, it’s really easy to fall into the trap of just using it, but point taken. Then main requirement is accurate moment to moment time. Using GPS as the master clock mostly makes sense there.
RossBencina 9mo ago
C++11 distinguishes system_clock from steady_clock. As you say, using system_clock is a bug.

Asmod4n 9mo ago

Does wall clock time matter for anything but logging? For everything else one could just create any form „time“ to keep stuff in sync, no?

tbrownaw 9mo ago
Isn't it also useful for checking validity periods for stuff like TLS certs or JWTs or Kerberos tickets?
- Asmod4n 9mo ago
  We could use some made up „time“ for that since it’s not made for human consumption just sync between different systems.
  - tbrownaw 9mo ago
    I suppose this is changing with TLS certs moving towards ephemerality, but they used to have an entry on someone's calendar for renewal.

NelsonMinar 9mo ago

On the flipside, clock sync for civilians has never been easier. Thanks to NTP any device with an Internet connection can pretty easily get time accurate to 1 second, often as little as 10 ms. All major consumer computers are preconfigured to sync time to one of several reliable NTP pools. This post is about more complicated synchronization for more demanding applications. And it's very good. I'm just marveling at how in my lifetime I from "no clock is ever set right" to assuming most anything was within a second of true time.

jasonwatkinspdx 9mo ago
At this point the only clock in my life that doesn't auto set is the one on my stove, and that's because I abhor internet connected kitchen appliances.
- amelius 9mo ago
  Same here. I wish there was an easy way around it (that doesn't require me to play sysadmin in my spare time).
  - jaggederest 9mo ago
    In the 80s my uncle had digital clocks that used an antenna to tune into the atomic clock time signal that (was/is?) broadcast nationwide. I've long wished that it was incorporated into stoves, microwaves, essentially everything that isn't an internet device (yet... sigh) Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of a pin these days.
    - jasonwatkinspdx 9mo ago
      > Sadly I think the actual antenna and hardware were relatively large since it's a long wave signal, but maybe with SDR it'll all fit on the head of a pin these days. Unfortunately there's no real way to cheat physics as far as shrinking a wavelength goes. With RF antennas about the best you can do is a major dimension 1/10th the frequency of interest.

shomp 9mo ago

Absolute synchronization impossible?? Challenge accepted.

nuccy 9mo ago
Nature (laws of physics) is agains you on this: it is in fact impossible for everyone. What is in sync for some observers can be out of sync for others (depends on where they are, i.e. gravity, and how they relatively move). See general and special relativity principle of simultaneity [1]. 1. https://en.wikipedia.org/wiki/Relativity_of_simultaneity https://en.wikipedia.org/wiki/Relativity_of_simultaneity
- Enginerrrd 9mo ago
  I think you just nerd-sniped me but I’m not convinced it’s impossible to assign a consistent ordering to events with relativistic separations. For starters, the spacetime interval between two events IS a Lorentz invariant quantity. That could probably be used to establish a universal order for timelike separations between events. I suspect that you could use a reference clock, like a pulsar or something to act as an event against which to measure the spacetime interval to other events, and use that for ordering. Any events separated by a light-like interval are essentially simultaneous to all observers under that measure. The problem comes for events with a space like or light like separation. In that case, the spacetime interval is still conserved, but I’m not sure how you assign order to them. Perhaps the same system works without modification, but I’m not sure.
  - tobias2014 9mo ago
    For any space-like event you can find reference frames where things happen in different order. For the time-like situation you described the order indeed exists within the cone, which is to say that causality exists.
    - Enginerrrd 9mo ago
      You can still order them with the spacetime interval compared to a reference event, even for space like separated events. It allows for differing elements of the set to share the same value but so does using time alone. It just also allows every observer to agree on the ordering. Bc Assigning a distance function to elements of a set is a common way to do that in fact. It doesn’t work with just a time coordinate or space coordinate, because that’s effectively a Euclidean metric. You just have to contend with a few nonintuitive aspects but it’s not so bad.

josephg 9mo ago

> When two transactions happen at nearly the same time on different nodes, the database must determine which happened first. If clocks are out of sync, the database might order them incorrectly, violating consistency guarantees. This is only true if you use wall clock time as part of your database’s consistency algorithm. Generally I think this is a huge mistake. It’s almost always much easier to swap to a logical clock - which doesn’t care about wall time. And then you don’t have to worry about ntp. The basic idea is this: event A happened before event B iff A (or something that happened after A) was observed by the node that generated B before B was generated. As a result, you end up with a dag of events - kind of like git. Some events aren’t ordered relative to one another. (We say, they happened concurrently). If you ever need a global order for all events, you can deterministically pick an arbitrary order for concurrent events by comparing ids or something. And this will give you a total order that will be the same on all peers. If you make database events work like this, time is a little more complex. (It’s a graph traversal rather than simple numbers). But as a result the system clock doesn’t matter. No need to worry about atomic clocks, skew, drift, monotonicity, and all of that junk. It massively simplifies your system design.

johnisgood 9mo ago
Related in many ways: https://www.erlang.org/docs/22/apps/erts/time_correction https://www.erlang.org/docs/22/apps/erts/time_correction Also I still remember having fun with the "Determine the order of events by saving a tuple containing monotonic time and a strictly monotonically increasing integer as follows" part.
b112 9mo ago
Unfortunately, some of us have to deal with things like billing, transaction timing to validate what a client's logs might have on their systems, and so on. My take on this is that second timing is close enough for this. And that all my internal systems need agree on the time. So if I'm off by 200ms or some blather from the rest of the world, I'm not overly concerned. I am concerned, however, if a random internal system is not synced to my own ntp servers. This doesn't mean I don't keep our servers synced, just that being off by some manner of ms doesn't bother me inordinately. And when it comes to timing of events, yes, auto-increment IDs or some such are easier to deal with.

dmazin 9mo ago

I highly recommend anyone to look up how PTP works and how it compares to NTP. Clock sync is very interesting. When I joined an HFT company, first thing I did was understand this stuff. We care about it a lot[1]. If you want a specific question to answer, answer this: why does PTP need hardware timestamping to achieve high precision (where the network card itself assigns timestamps to packets, rather than having the kernel do it as part of TCP/IP processing)? If we use software timestamps, why can we do microsecond precision at best? If you understand this, it goes a very long way to understanding the core ideas behind precise clock sync. Once you have a solid understanding of PTP, look into White Rabbit. They’re able to sync two clocks with sub-ns precision. In case that isn’t obvious, that is absolutely insane. [1] So do a lot of people. For example audio engineers. Once, an audio engineer absolutely talked my ear off about ptp. I had no idea that audio people understood clock sync so well but they do!

RossBencina 9mo ago
> So do a lot of people. For example audio engineers. Indeed. PTP (various, not-necessarily compatible, versions) is at the core of modern ethernet-based audio networking: Dante (proprietary, PTP: IEEE 1588 v1), AVB (IEEE standard, PTP: 802.1AS), AES67 (AES standard, PTP: IEEE 1588 v2). And now the scope of the AVB protocol stack has been expanded to TSN for industrial and automotive time sensitive network applications.
- dmazin 9mo ago
  Yeah, the audio engineer then talked my ear off about networking!
baby_souffle 9mo ago
Not just audio, anybody in the live events / production space needs all equipment marching in lock step.
- 8n4vidtmkvmk 9mo ago
  If it's for an event, can they not bring all the devices together in close proximity and sync them somehow? That at least removes network delays

didgetmaster 9mo ago

Reminds me of the old saying: 'If you have just one watch/clock, then you always know what time it is; but if you have two of them, then you are never sure!'

pdeva1 9mo ago

AWS has the Google TrueTime equivalent precision clock available for public use[1] which makes this problem much easier to solve now. Auora DSQL uses it. Even third party db's like YugabyteDb make use of it. [1] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time...

gdcohen 9mo ago

Take a look at Exploiting a Natural Network Effect for Scalable, Fine-grained Clock Synchronization - https://www.usenix.org/conference/nsdi18/presentation/geng https://www.usenix.org/conference/nsdi18/presentation/geng (commercial version by the same authors - clockwork.io).

sreekanth850 9mo ago

In physics, time is local and relative, independent events don’t need a global ordering. Distributed databases shouldn’t require one either. The idea of a single global time comes from 1980s single-node database semantics, where serializability implied one universal execution order. When that model was lifted into distributed systems, researchers introduced global clocks and timestamp coordination to preserve those guarantees, not because distributed systems fundamentally need it. It’s time we rethink this., Only operations that touch the same piece of data require ordering. Everything else should follow causality like the physical universe, independent events don’t need to agree on sequence, only dependent ones do. Global clocks exist because some databases forced serializable cross-object transactions onto distributed systems, not because nature requires it. Edit: I welcome for a discussion with people who disagree and downvote.

otterley 9mo ago
You can’t be certain that any given mutating operation you perform now won’t be relied upon for some future operation, unless the two operations are performed in entirely different domains of data. Even “not touching (by which I assume you mean mutating) the same data” isn’t enough. If I update A in thread 0 from 1 to 2, then I update B in thread 1 to the value of A+1, then the value of B could end up being 2 or 3, depending on whether the update of A reached thread 1.
- sreekanth850 9mo ago
  In distributed systems, dependencies flow forward, not backward. Causal dependency only exists when an operation actually references earlier state. If B = A+1, then yes, B is causally dependent on A and they must share an order. But that dependency is created by the application logic, not assumed globally in advance. We shouldn’t impose a universal timeline just because some future operation might depend on some past one. Dependencies should be explicit and local: if two operations interact, they share a causal scope; if they don’t, they shouldn’t pay the cost of coordination.
  - otterley 9mo ago

b112 9mo ago

The article doesn't cover the inane stupid that is: * NTP pool server usage requires using DNS * people have DNSSEC setup, which requires accurate time or it fails So if your clock is off, you cannot lookup NTP pool servers via DNS, and therefore cannot set your clock. This sheer stupid has been discussed with package maintainers of major distros, with ntpsec, and the result is a mere shrug. Often, the answer is "but doesn't your device have a battery backed clock?", which is quite unhelpful. Many devices (routers, IOT devices, small boards, or older machines, etc) don't have a battery backed clock, or alternatively the battery may just have died. Beyond that, the ntpsec codebase has a horrible bug where if DNS is not available when ntpsec starts, pool server addresses are never, ever retried. So if you have a complete power-fail in a datacentre rack, and your firewalls take a little longer to boot than your machines, you'll have to manually restart ntpsec to even get it to ever sync. When discussing this bug the ntpsec lads were confused that DNS might not exist at times. Long story short, make sure you aren't using DNS in any capacity, in NTP configs, and most especially in ntpsec configs. One good source is just using the IPs provided by NIST. Pool servers may seem fine, but I'd trust IPs assigned to NIST to exist longer than any DNS anyhow. EG, for decades.

ectospheno 9mo ago
I gave up and just got a gps ntp server. If you have a wood house then you can even get away with an inside antenna.

simonebrunozzi 9mo ago

Back when I was studying computer science, I was taking the OS exam and the part about Lamport timestamp [0] was optional, but I had studied it because I loved it. When I mentioned it to my professor, he was so happy to hear something new that day that he asked me to describe it in details. This was the year 2001. Many years later, in 2020, I ended up living in San Francisco, and I had the fortune to meet Leslie Lamport after I sent him a cold email. Lovely and smart guy. This is the text of the first part of that email, just for your curiosity: Hey Leslie! You have accompanied me for more than 20 years. I first met your name when studying Lamport timestamps. And then on, and on, and on, up to a few minutes ago, when I realized that you are also behind the paper and the title of "Byzantine Generals problem", renamed after the "Albanian" generals to the suggestion of Jack Goldberg. Who is he? [1] [0]: https://en.wikipedia.org/wiki/Lamport_timestamp https://en.wikipedia.org/wiki/Lamport_timestamp [1]: Jack Goldberg (now retired) was a computer scientist and Lamport's manager at SRI.

eatsome 9mo ago

For an article written about time, I would have thought there'd be a timestamp on the blog post. Just something to think about if someone stumbles upon this in a few years.

amiune 9mo ago

Related: https://632nm.com/episodes/why-syncing-atomic-clocks-is-virtually-impossible-judah-levine-on-utc https://632nm.com/episodes/why-syncing-atomic-clocks-is-virt... As a teacher I love the way Judah Levine explains

user3939382 9mo ago

That’s because neither discrete time nor synchronous network comms exist.

layer8 9mo ago

> The good news is that the International Bureau of Weights and Measures has decided to stop adding leap seconds by 2035. This is not entirely correct. What has been agreed is to allow deviations of more than one second after 2035, so that clocks have to be adjusted less frequently (on the order of every 50-100 years is the intention). However, the allowable deviation, and how to adjust clocks when it is exceeded, has yet to be decided.

danzheng 9mo ago

One thing missing in the blogpost is in practice you see many large orgs, especially in finance, living with multiple time domains. For example, on-prem trading systems almost always use PTP or PPS for sub-microsecond timestamping, often on dedicated networks to reduce jitter (for meeting regulatory requirements like MiFID II and CAT) while the rest of their infra (in on-prem and cloud) just runs NTP for millisecond-class sync. Both protocols are fundamentally sensitive to network conditions — the mean offset may look fine, but outliers due to congestion/jitter can be very poor. The consequence of having multiple time domains is pretty painful when you need to reconcile logs or transaction histories across systems with different sync accuracy. Millisecond NTP logs and sub-microsecond PTP logs don’t line up cleanly, so correlating events end-to-end can become guesswork rather than deterministic ordering. If you want reliable cross-system telemetry and audit trails, you'll need a single, high-accuracy time sync approach across your whole stack.

sureshvoz 9mo ago

This is a great breakdown, and it’s worth noting that we are hitting a "microsecond wall" in modern GPU clusters that makes standard NTP effectively obsolete. In distributed training (LLMs), the bottleneck is no longer just disk I/O or CPU cycles—it’s the "straggler problem" during collective communication (like All-Reduce). When you’re running on 400Gbps+ RoCE (RDMA over Converged Ethernet) networks, the network "wire time" is often lower than the clock jitter on a standard Linux kernel. If your clocks are skewed by even 2-3 milliseconds, your telemetry becomes essentially useless. It looks like packets are arriving before they were sent, or worse, your profiling tools can’t accurately pinpoint which GPU is stalling the rest of the 16,384-node fleet. We’ve reached a point where microsecond-accurate clocks isn't just a requirement for HFT firms; it’s becoming the baseline for anyone trying to keep $100s of millions of NVidia GPUs from idling while they wait for a collective sync.

perryizgr8 9mo ago
If you have network infrastructure that supports 400G I'm pretty sure it has solid PTP built in. And as far as I remember from my networking days setting it up is almost as simple as setting up NTP, you just need a single machine with a GPS lock.

j_seigh 9mo ago

The comments about HFT needing tightly synchronized clocks got me thinking. Back in the day, way back in the 80's, IBM replaced the VM with VMXA. VM could trap and emulate all the important instructions since they were privileged instructions except one, the STCK (store clock) instruction. So virtual machines couldn't set their virtual clocks so they were always in sync. VMXA used new hw features that let you set the virtual clock. You could specify an offset to the system clock. But some of IBM's biggest customers depended on all the virtual machines clocks always being in sync. So VMXA had to add an option to disallow setting the clock for specified virtual machines. Except all of development knew how trivial it was to trap or modify the STCK's to produce a timestamp of you choosing. This was before it was common knowledge the client code should never be trusted. But nobody enlightened IBM corporate management. It was a serious career limiting move at IBM. It didn't matter if you were right. So I'm pretty sure some serious fortunes were made as a result. So the question for HFT is; are they using and trusting client timestamps, or are the timestamps being generated on the market maker's servers? If the latter, how would the customer know?

jmpman 9mo ago

Back in the early 2000s I was programming on an IBM AIX server. Multicore, maybe multiprocessor and within the same machine, the clocks were skewed between the processors. If you’d dispatch a process, and then check its outstanding running time, it would differ depending upon which processor you’d check from, and of course it was a signed type, and then we would get negative values, which sent our code down the wrong path.