3 ms·

If your get_atomic() isn't also capturing or resetting an internal hardware timer atomically it's kinda broken code anyway. It's important to remember that you'

by kurthr 4y ago

If your get_atomic() isn't also capturing or resetting an internal hardware timer atomically it's kinda broken code anyway. It's important to remember that you're getting data from the firmware of an atomic clock (or GPS receiver) and that they will go to great lengths to compensate for latency variations so not doing it in the sampling code would be nuts (or just not using isochronous coms). Then the local clock/timer can be calibrated from the that and you can have near clock-cycle correct timing. Typically they only send out 1 pulse per second so anything finer than that is locally interpolated. We used 1ms internal intervals with an accuracy of ~1ppm. Last time I worked on something like this was 20 years ago, but it was a Pentium box. The clock firmware to converted a 5MHz sinewave output to 1Pps, but could also output a synthetic UTC through LAN in cases where there was no other connectivity. I guess I'm saying this is really specialized code for whatever hardware/OS you're using. (edit) Here's some documentation for a very similar box: https://www.manualslib.com/manual/1281621/Symmetricom-5071a.html?page=27#manual https://www.manualslib.com/manual/1281621/Symmetricom-5071a....

jrockway 4y ago
The GPS clock is really used to provide long-term stability for the computer's internal oscillator, which provides fine short-term stability. Your calls to gettimeofday() are handled by the internal oscillator, not by asking the GPS unit directly. Additionally, NTP daemons are impressive in their ability to get the "right answer" on current internal oscillator frequency as affected by the entire system. With 86400 very accurate data samples per day, you can do a lot! In a distributed system, the absolute value of the time is important, since that's what your transaction timestamps are based on. There is a lot to do to ensure that your time offset from UTC ends up being correct; antenna cable length compensation, oscillator quantization compensation, etc. These are not strictly necessary for Spanner but can make microsecond-level differences which are quite noticeable. "The API directly exposes clock uncertainty, and the guarantees on Spanner’s timestamps depend on the bounds that the implementation provides. If the uncertainty is large, Spanner slows down to wait out that uncertainty. Google’s cluster-management software provides an implementation of the TrueTime API. This implementation keeps uncertainty small (generally less than 10ms) by using multiple modern clock references (GPS and atomic clocks)." Basically, there is some tradeoff between clock synchronization perfection and transaction processing speed that can be made. You can build dedicated hardware that's clocked by a GPSDO and provide an API to get hardware timestamps, and maybe process transactions a little faster. Your good old C++ program running on Linux with an internal oscillator adjusted by NTP with a GPS PPS input still beats communicating between datacenters to agree on event ordering, though. Light is slow!
- eternalban 4y ago
  > In a distributed system, the absolute value of the time is important Only for a co-variant data cohort is total temporal order necessary for correctness. It doesn't need to be system wide. Distinct (data independent) process groups can be partially ordered. There really is no such thing as "now" outside of a shared frame (or intersecting frames) of observation.
hansvm 4y ago
All I was pointing out is that worst-case latency on a throughput-optimized OS can be unbounded, and it's not totally unexpected to see 1ms+ delays between those two lines of code, even if the clock implementation is flawless. Even in a RTOS you have jitter from cache misses and whatnot, just to a lesser degree. Code sensitive to clocks not being correct is often sensitive no matter how small the deviance is, and smaller deviations simply make bugs less frequent. Treating that point-in-time estimate as anything other than an estimate from the recent past can lead to code that looks flawless but occasionally breaks.

by kurthr 4y ago

jrockway 4y ago

The GPS clock is really used to provide long-term stability for the computer's internal oscillator, which provides fine short-term stability. Your calls to gettimeofday() are handled by the internal oscillator, not by asking the GPS unit directly. Additionally, NTP daemons are impressive in their ability to get the "right answer" on current internal oscillator frequency as affected by the entire system. With 86400 very accurate data samples per day, you can do a lot! In a distributed system, the absolute value of the time is important, since that's what your transaction timestamps are based on. There is a lot to do to ensure that your time offset from UTC ends up being correct; antenna cable length compensation, oscillator quantization compensation, etc. These are not strictly necessary for Spanner but can make microsecond-level differences which are quite noticeable. "The API directly exposes clock uncertainty, and the guarantees on Spanner’s timestamps depend on the bounds that the implementation provides. If the uncertainty is large, Spanner slows down to wait out that uncertainty. Google’s cluster-management software provides an implementation of the TrueTime API. This implementation keeps uncertainty small (generally less than 10ms) by using multiple modern clock references (GPS and atomic clocks)." Basically, there is some tradeoff between clock synchronization perfection and transaction processing speed that can be made. You can build dedicated hardware that's clocked by a GPSDO and provide an API to get hardware timestamps, and maybe process transactions a little faster. Your good old C++ program running on Linux with an internal oscillator adjusted by NTP with a GPS PPS input still beats communicating between datacenters to agree on event ordering, though. Light is slow!

eternalban 4y ago
> In a distributed system, the absolute value of the time is important Only for a co-variant data cohort is total temporal order necessary for correctness. It doesn't need to be system wide. Distinct (data independent) process groups can be partially ordered. There really is no such thing as "now" outside of a shared frame (or intersecting frames) of observation.

hansvm 4y ago

All I was pointing out is that worst-case latency on a throughput-optimized OS can be unbounded, and it's not totally unexpected to see 1ms+ delays between those two lines of code, even if the clock implementation is flawless. Even in a RTOS you have jitter from cache misses and whatnot, just to a lesser degree. Code sensitive to clocks not being correct is often sensitive no matter how small the deviance is, and smaller deviations simply make bugs less frequent. Treating that point-in-time estimate as anything other than an estimate from the recent past can lead to code that looks flawless but occasionally breaks.