8 ms·

Living Without Atomic Clocks (2016)

94 points by bobbiechen 7y ago

kens 7y ago
CuriousMarc made a video last week where he uses a vintage HP 5061A atomic clock and explains how it works. It's interesting to see that atomic clocks aren't just some giant laboratory thing, but a product you could buy, even in the 1960s. https://www.youtube.com/watch?v=eOti3kKWX-c https://www.youtube.com/watch?v=eOti3kKWX-c
- tyingq 7y ago
  CDW has a Cesium reference clock for $92k. https://m.cdw.com/product/microsemi-5071a-high-performance-tube-cesium-clock-with-48-vdc-psu/5654336?enkwrd=Cesium https://m.cdw.com/product/microsemi-5071a-high-performance-t... Pretty similar form factor.
  - vibrolax 7y ago
    I believe this is the direct successor to the HP 5061A. HP spun off Agilent, who sold the time/frequency product division to Symmetricom, who was then acquired by Microsemi. It seems that the inflation-adjusted price of the Microsemi 5071A is about the same as the ~1970 catalog price of the HP 5061.
- michaelt 7y ago
  You can buy a used rubidium frequency standard on ebay [1] for less than $200. Of course, it turns out there aren't that many home applications for an atomic clock, other than collecting precision metrology equipment. And if you're running a data centre and want higher precision than NTP, chances are you'll choose a PTP grandmaster clock at the high end, or a GPS receiver with a 1PPS output at the low end, in rather than buying second-hand parts from ebay. [1] https://www.ebay.com/sch/i.html?_nkw=rubidium https://www.ebay.com/sch/i.html?_nkw=rubidium
  - segfaultbuserr 7y ago
    > Of course, it turns out there aren't that many home applications for an atomic clock For all the people who have a rubidium standard at home, 70% are using it for electronic lab, amateur radio, or NTP at home. But the remaining 30% is another large userbase too - audiophiles. Many audiophiles claim jitter and phase noise in the clock signal significantly affects audio quality, and the most extreme audiophiles use a lab grade frequency standard, rubidium or better, and feed it to all their Hi-Fi gears. P.S: Given all the factors that affect audio quality, is the phase noise from a PLL synthesizer of reasonable quality really a major factor? Even if it is, does a rubidium standard really have any benefit beyond diminishing return over a crystal oven? Well, of course, these are the questions that are never answered by audiophiles. > You can buy a used rubidium frequency standard on ebay There is a continuous source of used rubidium standard coming from retired telecommunication and lab equipment, they are the cheapest atomic standard available. The catch is that the rubidium inside the discharge lamp will eventually get depleted during operation, usually within 10 years, once its life ends, it's useless and needs to be rebuilt completely by the manufacturer. So read the manufacturing date if you are going to power it 7x24. The real expensive ones are the Cesium Standard, such as the HP 5061A. Recently, the Hydrogen Standard also saw some uses.
    - JackMcMack 7y ago
      The rubidium does not deplete (it's a closed system), but does settle on the glass. Supposedly you can heat the glass to turn all the rubidium into gas again, giving you theoretically infinite lifetime. I picked up a used PRS-10 on ebay for $250. It's a small but great oscillator, still made today. The manufacturer SRS claims 20 years lifetime, and mine has had 13 years of runtime, so I should be able to get 7 years out of it hopefully. The 11W1 DSub connector they used on the other hand is stupidly expensive. Cesium oscillators on the other hand do deplete their cesium (by design), as mentioned by CuriousMarc in the video. The tube in his unit was probably replaced at least twice, at a cost of >80k. The high performance tubes fare even worse because they "burn" through their cesium at a 3x rate. The 5071 successor by HP-Agilent-Symmetricom-Microsemi is the most frequently used COTS clock that contributes to UTC.
- bobbiechen 7y ago
  Yeah I had always thought they would be incredibly expensive. The Spanner paper (which is how I ended up at the posted link) contains this amusing line - An atomic clock is not that expensive: the cost of an Armageddon master is of the same order as that of a GPS master http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf http://static.googleusercontent.com/media/research.google.co...
hinkley 7y ago
My first long-term job involved collecting telemetry data from equipment and displaying it. The display was technically challenging, but in the meaty way that developers often relish. The clock skew, however, was not. For liability purposes it was sometimes necessary to know if Event A happens before Event B, which means you have to normalize all of the events across time zones and then correct for drift too. That experience and a bunch of others (including statistics classes, and studying the Java Memory Model, which other languages have borrowed or stolen) have left me with a lingering doubt about how we record distributed activities. I really kind of feel like we [all] need a new data model [like the one they mention here] where dependent events are recorded in that way. I don't know exactly what that would look like, but I think it would help a great deal in consensus situations where you have to resolve a conflict, or even just for displaying a sequence of events in proper order. It feels like we keep trying to get the exact nanosecond when something happens, but the only thing I ever see humans use that information for is to reverse engineer a sequence of events that resulted in a peculiar state in the system. [edit: tie-in to article]
- dinosaurdynasty 7y ago
  Sounds a lot like logical clocks https://en.wikipedia.org/wiki/Logical_clock https://en.wikipedia.org/wiki/Logical_clock, especially something like a vector clock https://en.wikipedia.org/wiki/Vector_clock https://en.wikipedia.org/wiki/Vector_clock.
- valenceelectron 7y ago
  >It feels like we keep trying to get the exact nanosecond when something happens, but the only thing I ever see humans use that information for is to reverse engineer a sequence of events that resulted in a peculiar state in the system. Lamport clocks do exist and can provide a partial or total order of events. Not sure if that would've helped with your problem but such algorithms do exist. But it feels like they are rarely used (subjective feeling).
  - dinosaurdynasty 7y ago
    It should be noted that the total ordering that Lamport clocks provide is fairly artificial (in fact it's best to say that there is no natural total ordering of events in a distributed system). Agreed, though you do see them a lot in CRDTs: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_.... Just about every non-trivial CRDT has something akin to a logical clock embedded inside it.
aidenn0 7y ago
I'm curious what the distribution of time offsets between servers running NTP is. Properly functioning NTP should maintain an offset well under 100ms, but what fraction of the time servers are "properly functioning" is a question I'm interested in knowing.
- thedance 7y ago
  This is really the key point, and the main thing that many people overlook. The purpose of TrueTime in Spanner is not gained by absolute accuracy. It's gained by knowing the clock uncertainty across all participants and affirmatively ejecting any participant with excessive uncertainty. The atomic clocks are totally besides the point.
  - hinkley 7y ago
    But the atomic clock is how they keep the throughput from crashing every time a corner case is encountered.
- hermitdev 7y ago
  PTP (Precision Time Protocol) can get you sub microsecond synchronization on a LAN. See: https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol Most recent spec is IEEE 1588-2019.
  - aidenn0 7y ago
    NTP is more than good enough to get 7ms synchronization on a LAN when working correctly, so the fact that PTP can do even better is perhaps uninteresting in light of the fact that NTP often fails to get anywhere near its theoretical performance (other comments mention clocks off by years).
- sethammons 7y ago
  Talking with ops folks at work, they have seen system clocks in our fleet (which run ntpd) suddenly report times off by _years_ and then go back to "accurate." We have thousands of nodes for the software I work on. We sometimes get negative times when comparing events, if only by a few seconds. You can't fully trust time in a distributed system.
thedance 7y ago
The latest entry in the surprising crowded genre: How can we copy Spanner, but without all that pesky correctness?
- jordanlewis 7y ago
  The post is from 2016.
  - thedance 7y ago
    You're right, this is the one that started it all. Sorry for being so jaded; I've been through two distinct efforts of "Spanner, but without thinking" in my recent career.
    - dang 7y ago
      Since you obviously know about this, it would be good to share some of what you know, so the rest of us can learn. Dismissive comments without information lower discussion quality, not to mention frustrate curious users. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
dang 7y ago
Discussed at the time: https://news.ycombinator.com/item?id=11120744 https://news.ycombinator.com/item?id=11120744
deleted 7y ago
[deleted]
- gok 7y ago
  "It’d be a showstopper to require an external dependency on specialized hardware for clock synchronization."
ckocagil 7y ago
It shouldn't be too expensive to build and deploy GPS disciplined oscillators based on commodity crystal oscillators (temperature compensated or ovenized). Only one would be needed for a whole datacenter, then local NTP would provide it to the LAN. No need for fancy telecoms grade Rubidium/Cesium stuff.
- jrockway 7y ago
  The fancy oscillators are for longer "holdover" time. It's essential that frequency not go out of spec for CDMA (etc.) networks even if the GPS signal is degraded for whatever reason, so they use oscillators with large holdover times to buy themselves an extra layer of redundancy. I don't know if you need that for distributed transactions or not. The logic as to whether or not the time is "good" is probably a large part of the complexity of this scheme. Better hardware makes the software simpler and vulnerable to less failure modes.
bitwize 7y ago
> In essence, it provides a means to absolutely order events, regardless of which distributed node an event originated at. Einstein said you can't absolutely order events -- and with widely enough distributed systems and small enough time quanta, sooner or later you're going to run into relativistic implications. Probably not a problem for most applications we currently have to deal with, but one day -- soon enough that we're already giving new protocols names like "Interplanetary File System" -- our databases will spread out among the stars, and how will we handle time and event ordering then?
- AmericanChopper 7y ago
  For most use cases, you’d just give up serializability and implement an eventually consistent model. For anything that requires something more strict, you just wait. Longer. Processing a check used to take several days. Processing an earth bank check from mars would need to take about half an hour on average. Though that’s perhaps not the best example, because even today we process checks using an eventual consistency model. Edit: A strict consistency earth > mars transaction would actually take longer, because 30 minutes is just an average round trip, and you’d need more than one.
  - daxfohl 7y ago
    Note that even with Google Cloud Spanner (last I checked anyway), the distributed consistency model was only available among machines within the same datacenter, as orchestrating transactions across regions would be latency prohibitive. I think you can get cross-region read replicas but that's all. (And I may even be mixing that up with some other cloud database). So interplanetary, yeah one planet is going to be the central transaction handler and everyone else is going to have to deal with it. So likely things that require transactions, you'll have to have enough cash in the bank in your Mars account if you want quick transactions, and expect wiring money from Earth to take a while. Coming back to the current day, if you need low-latency global transactions just here on Earth, then that's probably how you'd have to design it. You prime each region with a certain account "limit" of whatever it is you're transacting to use in local transactions, and when that limit gets low you transact some more from your central data store back into your regional account, or something along those lines. It'd be a two-stage thing.
bogomipz 7y ago
A long time ago I worked in a finance/trading environment and we had GPS clock receivers in our datacenters that synced from satellites. While they weren't that expensive I would imagine at Google's scale they would might be. Is there a reason Google wouldn't have used this same technique if it achieves the same accuracy?
- scottlamb 7y ago
  iirc, one big reason is leap second handling. The closed-source GPS receivers' firmware tend to behave strangely around leap seconds, in a way that contradicts their documentation. And their behavior is untestable in advance: a global satellite network for time synchronization is the ultimate un-mockable time bomb input.
  - bogomipz 7y ago
    Thanks but isn't that a reason to forgo using GPS receivers? Google obviously decided to use them however.
    - scottlamb 7y ago
      Oh, sorry. I was distracted and misread your comment. I was trying to answer the question of "why an atomic clock rather than a GPS receiver" (and Google's "Armageddon masters" have their own atomic clocks), but I see now you were asking "why not use Cockroach's technique rather than any specialized hardware". I think one answer is in this blog post: > A simple statement of the contrast between Spanner and CockroachDB would be: Spanner always waits on writes for a short interval, whereas CockroachDB sometimes waits on reads for a longer interval. How long is that interval? Well it depends on how clocks on CockroachDB nodes are being synchronized. Using NTP, it’s likely to be up to 250ms. Not great, but the kind of transaction that would restart for the full interval would have to read constantly updated values across many nodes. In practice, these kinds of use cases exist but are the exception.
      - bogomipz 7y ago

94 points by bobbiechen 7y ago

kens 7y ago

CuriousMarc made a video last week where he uses a vintage HP 5061A atomic clock and explains how it works. It's interesting to see that atomic clocks aren't just some giant laboratory thing, but a product you could buy, even in the 1960s. https://www.youtube.com/watch?v=eOti3kKWX-c https://www.youtube.com/watch?v=eOti3kKWX-c

tyingq 7y ago
CDW has a Cesium reference clock for $92k. https://m.cdw.com/product/microsemi-5071a-high-performance-tube-cesium-clock-with-48-vdc-psu/5654336?enkwrd=Cesium https://m.cdw.com/product/microsemi-5071a-high-performance-t... Pretty similar form factor.
- vibrolax 7y ago
  I believe this is the direct successor to the HP 5061A. HP spun off Agilent, who sold the time/frequency product division to Symmetricom, who was then acquired by Microsemi. It seems that the inflation-adjusted price of the Microsemi 5071A is about the same as the ~1970 catalog price of the HP 5061.
michaelt 7y ago
You can buy a used rubidium frequency standard on ebay [1] for less than $200. Of course, it turns out there aren't that many home applications for an atomic clock, other than collecting precision metrology equipment. And if you're running a data centre and want higher precision than NTP, chances are you'll choose a PTP grandmaster clock at the high end, or a GPS receiver with a 1PPS output at the low end, in rather than buying second-hand parts from ebay. [1] https://www.ebay.com/sch/i.html?_nkw=rubidium https://www.ebay.com/sch/i.html?_nkw=rubidium
- segfaultbuserr 7y ago
  > Of course, it turns out there aren't that many home applications for an atomic clock For all the people who have a rubidium standard at home, 70% are using it for electronic lab, amateur radio, or NTP at home. But the remaining 30% is another large userbase too - audiophiles. Many audiophiles claim jitter and phase noise in the clock signal significantly affects audio quality, and the most extreme audiophiles use a lab grade frequency standard, rubidium or better, and feed it to all their Hi-Fi gears. P.S: Given all the factors that affect audio quality, is the phase noise from a PLL synthesizer of reasonable quality really a major factor? Even if it is, does a rubidium standard really have any benefit beyond diminishing return over a crystal oven? Well, of course, these are the questions that are never answered by audiophiles. > You can buy a used rubidium frequency standard on ebay There is a continuous source of used rubidium standard coming from retired telecommunication and lab equipment, they are the cheapest atomic standard available. The catch is that the rubidium inside the discharge lamp will eventually get depleted during operation, usually within 10 years, once its life ends, it's useless and needs to be rebuilt completely by the manufacturer. So read the manufacturing date if you are going to power it 7x24. The real expensive ones are the Cesium Standard, such as the HP 5061A. Recently, the Hydrogen Standard also saw some uses.
  - JackMcMack 7y ago
    The rubidium does not deplete (it's a closed system), but does settle on the glass. Supposedly you can heat the glass to turn all the rubidium into gas again, giving you theoretically infinite lifetime. I picked up a used PRS-10 on ebay for $250. It's a small but great oscillator, still made today. The manufacturer SRS claims 20 years lifetime, and mine has had 13 years of runtime, so I should be able to get 7 years out of it hopefully. The 11W1 DSub connector they used on the other hand is stupidly expensive. Cesium oscillators on the other hand do deplete their cesium (by design), as mentioned by CuriousMarc in the video. The tube in his unit was probably replaced at least twice, at a cost of >80k. The high performance tubes fare even worse because they "burn" through their cesium at a 3x rate. The 5071 successor by HP-Agilent-Symmetricom-Microsemi is the most frequently used COTS clock that contributes to UTC.
bobbiechen 7y ago
Yeah I had always thought they would be incredibly expensive. The Spanner paper (which is how I ended up at the posted link) contains this amusing line - An atomic clock is not that expensive: the cost of an Armageddon master is of the same order as that of a GPS master http://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf http://static.googleusercontent.com/media/research.google.co...

hinkley 7y ago

My first long-term job involved collecting telemetry data from equipment and displaying it. The display was technically challenging, but in the meaty way that developers often relish. The clock skew, however, was not. For liability purposes it was sometimes necessary to know if Event A happens before Event B, which means you have to normalize all of the events across time zones and then correct for drift too. That experience and a bunch of others (including statistics classes, and studying the Java Memory Model, which other languages have borrowed or stolen) have left me with a lingering doubt about how we record distributed activities. I really kind of feel like we [all] need a new data model [like the one they mention here] where dependent events are recorded in that way. I don't know exactly what that would look like, but I think it would help a great deal in consensus situations where you have to resolve a conflict, or even just for displaying a sequence of events in proper order. It feels like we keep trying to get the exact nanosecond when something happens, but the only thing I ever see humans use that information for is to reverse engineer a sequence of events that resulted in a peculiar state in the system. [edit: tie-in to article]

dinosaurdynasty 7y ago
Sounds a lot like logical clocks https://en.wikipedia.org/wiki/Logical_clock https://en.wikipedia.org/wiki/Logical_clock, especially something like a vector clock https://en.wikipedia.org/wiki/Vector_clock https://en.wikipedia.org/wiki/Vector_clock.
valenceelectron 7y ago
>It feels like we keep trying to get the exact nanosecond when something happens, but the only thing I ever see humans use that information for is to reverse engineer a sequence of events that resulted in a peculiar state in the system. Lamport clocks do exist and can provide a partial or total order of events. Not sure if that would've helped with your problem but such algorithms do exist. But it feels like they are rarely used (subjective feeling).
- dinosaurdynasty 7y ago
  It should be noted that the total ordering that Lamport clocks provide is fairly artificial (in fact it's best to say that there is no natural total ordering of events in a distributed system). Agreed, though you do see them a lot in CRDTs: https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_.... Just about every non-trivial CRDT has something akin to a logical clock embedded inside it.

aidenn0 7y ago

I'm curious what the distribution of time offsets between servers running NTP is. Properly functioning NTP should maintain an offset well under 100ms, but what fraction of the time servers are "properly functioning" is a question I'm interested in knowing.

thedance 7y ago
This is really the key point, and the main thing that many people overlook. The purpose of TrueTime in Spanner is not gained by absolute accuracy. It's gained by knowing the clock uncertainty across all participants and affirmatively ejecting any participant with excessive uncertainty. The atomic clocks are totally besides the point.
- hinkley 7y ago
  But the atomic clock is how they keep the throughput from crashing every time a corner case is encountered.
hermitdev 7y ago
PTP (Precision Time Protocol) can get you sub microsecond synchronization on a LAN. See: https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol Most recent spec is IEEE 1588-2019.
- aidenn0 7y ago
  NTP is more than good enough to get 7ms synchronization on a LAN when working correctly, so the fact that PTP can do even better is perhaps uninteresting in light of the fact that NTP often fails to get anywhere near its theoretical performance (other comments mention clocks off by years).
sethammons 7y ago
Talking with ops folks at work, they have seen system clocks in our fleet (which run ntpd) suddenly report times off by _years_ and then go back to "accurate." We have thousands of nodes for the software I work on. We sometimes get negative times when comparing events, if only by a few seconds. You can't fully trust time in a distributed system.

thedance 7y ago

The latest entry in the surprising crowded genre: How can we copy Spanner, but without all that pesky correctness?

jordanlewis 7y ago
The post is from 2016.
- thedance 7y ago
  You're right, this is the one that started it all. Sorry for being so jaded; I've been through two distinct efforts of "Spanner, but without thinking" in my recent career.
  - dang 7y ago
    Since you obviously know about this, it would be good to share some of what you know, so the rest of us can learn. Dismissive comments without information lower discussion quality, not to mention frustrate curious users. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

dang 7y ago

Discussed at the time: https://news.ycombinator.com/item?id=11120744 https://news.ycombinator.com/item?id=11120744

deleted 7y ago

[deleted]

gok 7y ago
"It’d be a showstopper to require an external dependency on specialized hardware for clock synchronization."

ckocagil 7y ago

It shouldn't be too expensive to build and deploy GPS disciplined oscillators based on commodity crystal oscillators (temperature compensated or ovenized). Only one would be needed for a whole datacenter, then local NTP would provide it to the LAN. No need for fancy telecoms grade Rubidium/Cesium stuff.

jrockway 7y ago
The fancy oscillators are for longer "holdover" time. It's essential that frequency not go out of spec for CDMA (etc.) networks even if the GPS signal is degraded for whatever reason, so they use oscillators with large holdover times to buy themselves an extra layer of redundancy. I don't know if you need that for distributed transactions or not. The logic as to whether or not the time is "good" is probably a large part of the complexity of this scheme. Better hardware makes the software simpler and vulnerable to less failure modes.

bitwize 7y ago

> In essence, it provides a means to absolutely order events, regardless of which distributed node an event originated at. Einstein said you can't absolutely order events -- and with widely enough distributed systems and small enough time quanta, sooner or later you're going to run into relativistic implications. Probably not a problem for most applications we currently have to deal with, but one day -- soon enough that we're already giving new protocols names like "Interplanetary File System" -- our databases will spread out among the stars, and how will we handle time and event ordering then?

AmericanChopper 7y ago
For most use cases, you’d just give up serializability and implement an eventually consistent model. For anything that requires something more strict, you just wait. Longer. Processing a check used to take several days. Processing an earth bank check from mars would need to take about half an hour on average. Though that’s perhaps not the best example, because even today we process checks using an eventual consistency model. Edit: A strict consistency earth > mars transaction would actually take longer, because 30 minutes is just an average round trip, and you’d need more than one.
- daxfohl 7y ago
  Note that even with Google Cloud Spanner (last I checked anyway), the distributed consistency model was only available among machines within the same datacenter, as orchestrating transactions across regions would be latency prohibitive. I think you can get cross-region read replicas but that's all. (And I may even be mixing that up with some other cloud database). So interplanetary, yeah one planet is going to be the central transaction handler and everyone else is going to have to deal with it. So likely things that require transactions, you'll have to have enough cash in the bank in your Mars account if you want quick transactions, and expect wiring money from Earth to take a while. Coming back to the current day, if you need low-latency global transactions just here on Earth, then that's probably how you'd have to design it. You prime each region with a certain account "limit" of whatever it is you're transacting to use in local transactions, and when that limit gets low you transact some more from your central data store back into your regional account, or something along those lines. It'd be a two-stage thing.

bogomipz 7y ago

A long time ago I worked in a finance/trading environment and we had GPS clock receivers in our datacenters that synced from satellites. While they weren't that expensive I would imagine at Google's scale they would might be. Is there a reason Google wouldn't have used this same technique if it achieves the same accuracy?

scottlamb 7y ago
iirc, one big reason is leap second handling. The closed-source GPS receivers' firmware tend to behave strangely around leap seconds, in a way that contradicts their documentation. And their behavior is untestable in advance: a global satellite network for time synchronization is the ultimate un-mockable time bomb input.
- bogomipz 7y ago
  Thanks but isn't that a reason to forgo using GPS receivers? Google obviously decided to use them however.
  - scottlamb 7y ago
    Oh, sorry. I was distracted and misread your comment. I was trying to answer the question of "why an atomic clock rather than a GPS receiver" (and Google's "Armageddon masters" have their own atomic clocks), but I see now you were asking "why not use Cockroach's technique rather than any specialized hardware". I think one answer is in this blog post: > A simple statement of the contrast between Spanner and CockroachDB would be: Spanner always waits on writes for a short interval, whereas CockroachDB sometimes waits on reads for a longer interval. How long is that interval? Well it depends on how clocks on CockroachDB nodes are being synchronized. Using NTP, it’s likely to be up to 250ms. Not great, but the kind of transaction that would restart for the full interval would have to read constantly updated values across many nodes. In practice, these kinds of use cases exist but are the exception.
    - bogomipz 7y ago