4 ms·
> However, because of clock drifts and/or assumptions around network time delays, timestamps from conventional clocks are not always mutually comparable, and th
by vlovich123 4y ago
> However, because of clock drifts and/or assumptions around network time delays, timestamps from conventional clocks are not always mutually comparable, and therefore events cannot be reliably ordered using timestamps from conventional clocks.
This isn’t quite right. CockroachDB and Yuggabyte both use conventional clocks to reliably order events. Spanner uses GPS to do so (conventional time stamp but maybe not technically a conventional clock).
- amelius 4y agoI suppose you can always order events by the time it takes for light to (physically or hypothetically) travel to a central server plus the server's local time.
- diarrhea 4y agoNetwork uncertainties like splits and delays will render that impossible.
- ketzu 4y agoYour post made me read this: https://www.cockroachlabs.com/blog/living-without-atomic-clocks/ https://www.cockroachlabs.com/blog/living-without-atomic-clo... on how spanner and cockroadDB do this. And the answes are surprisingly simple, but with consequences attached. * Spanner creates very tight bounds on the clock synchronization (through atomic clocks/GPS) and ... just waits out the length of the bound. * CockroachDB seems to do lamport-clocks-with-real-timestamps for linearizability (track the highest seen timestamp for causality chains). For preventing consistency violations with reads, they also track the bounds on the clock and potentially attempt to read again. So they approve of the overall message (of not just comparing conventional timestamps) but work around those using the uncertainty of the clocks they have available. My professor used to introduce logical clocks with "as we usually can't use atomic clocks on all nodes" because of that.
- hansvm 4y agoThe "as we usually can't use atomic clocks on all nodes" constraint is more common than you might think. Even code as simple as ts = get_atomic_time_with_no_delay_because_this_node_has_an_atomic_clock(); do_stuff_with_time(ts); is really broken if you have to wait for garbage collection, wait for your process/thread/coroutine/... to be scheduled, wait for a nearly invisible pause as a cache line is refreshed, .... The time you're using the atomic timestamp is at some future moment, and if that delay matters for your algorithm you need to track your uncertainty just like Spanner/Cockroach do (you just get significant benefits because the uncertainty is low).
- kurthr 4y agoIf your get_atomic() isn't also capturing or resetting an internal hardware timer atomically it's kinda broken code anyway. It's important to remember that you're getting data from the firmware of an atomic clock (or GPS receiver) and that they will go to great lengths to compensate for latency variations so not doing it in the sampling code would be nuts (or just not using isochronous coms). Then the local clock/timer can be calibrated from the that and you can have near clock-cycle correct timing. Typically they only send out 1 pulse per second so anything finer than that is locally interpolated. We used 1ms internal intervals with an accuracy of ~1ppm. Last time I worked on something like this was 20 years ago, but it was a Pentium box. The clock firmware to converted a 5MHz sinewave output to 1Pps, but could also output a synthetic UTC through LAN in cases where there was no other connectivity. I guess I'm saying this is really specialized code for whatever hardware/OS you're using. (edit) Here's some documentation for a very similar box: https://www.manualslib.com/manual/1281621/Symmetricom-5071a.html?page=27#manual https://www.manualslib.com/manual/1281621/Symmetricom-5071a....
- jrockway 4y agoThe GPS clock is really used to provide long-term stability for the computer's internal oscillator, which provides fine short-term stability. Your calls to gettimeofday() are handled by the internal oscillator, not by asking the GPS unit directly. Additionally, NTP daemons are impressive in their ability to get the "right answer" on current internal oscillator frequency as affected by the entire system. With 86400 very accurate data samples per day, you can do a lot! In a distributed system, the absolute value of the time is important, since that's what your transaction timestamps are based on. There is a lot to do to ensure that your time offset from UTC ends up being correct; antenna cable length compensation, oscillator quantization compensation, etc. These are not strictly necessary for Spanner but can make microsecond-level differences which are quite noticeable. "The API directly exposes clock uncertainty, and the guarantees on Spanner’s timestamps depend on the bounds that the implementation provides. If the uncertainty is large, Spanner slows down to wait out that uncertainty. Google’s cluster-management software provides an implementation of the TrueTime API. This implementation keeps uncertainty small (generally less than 10ms) by using multiple modern clock references (GPS and atomic clocks)." Basically, there is some tradeoff between clock synchronization perfection and transaction processing speed that can be made. You can build dedicated hardware that's clocked by a GPSDO and provide an API to get hardware timestamps, and maybe process transactions a little faster. Your good old C++ program running on Linux with an internal oscillator adjusted by NTP with a GPS PPS input still beats communicating between datacenters to agree on event ordering, though. Light is slow!
- pookah 4y agoWhen a system overheats the CPU can speed up and the clocks drift and if you're producing volumes of events you will get corruption. Wouldn't ever use a system clock... You can write a hybrid logical clock in about forty lines of code that can scale as much as these wildly expensive appliances that these big tech companies shill.
- lanstin 4y agoThis is why reading the original Lamport paper is useful. This distributed nature of things isn’t due to bad clocks or software or anything. It is an irreducible part of the physics of our world, with relativity. Perfect clocks won’t help with space like separations and different velocities or gravitational force. You have to build communication into it. The causality relationship is inherently a partial order.
- cleansingfire 4y agoOriginal Lamport paper here https://lamport.azurewebsites.net/pubs/time-clocks.pdf https://lamport.azurewebsites.net/pubs/time-clocks.pdf
- lanstin 4y agothanks. this is a profoundly useful paper to help one being able to think about distributed systems.
- libraryofbabel 4y agoThis isn’t really true: it’s not got anything to do with relatively specifically, it’s just because of the finite and varying time for messages to travel between points in a distributed system. So it’s about physics but not about relativistic physics. A Newtonian universe with finite speed of light would have the same phenomenon. That said, the analogy with causality in relativity and light cones etc is useful, and it’s no coincidence that Lamport wrote a book on general relativity before he turned to distributed systems.
- magicalhippo 4y ago> A Newtonian universe with finite speed of light would have the same phenomenon. Isn't the whole point of special relativity that a Newtonian universe with a finite speed of light isn't Newtonian?
- libraryofbabel 4y ago
- dikei 4y agoIt really depends on how you define "reliability". If a system never orders events wrong but will immediately stops processing events when enough number of nodes have their clocks out-of-synced, which is the case with all systems using conventional clocks, can it still be considered reliable?
- lanstin 4y agoonly if you have a good ops team on 24x7 pager duty