5 ms·

Google Spans Planet with GPS-Powered Database (2012)

36 points by sds111 9y ago

mehrdada 9y ago
(2012)
ars 9y ago
This article explains exactly nothing. You can replace the word TrueTime, with "Magic" and learn exactly as much.
- sumitgt 9y ago
  The recent paper on Spanner has some details of how it is used if you are interested. https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf https://static.googleusercontent.com/media/research.google.c...
itronitron 9y ago
you can make one of these yourself now if you need highly accurate (but not completely accurate all the time) time stamps... https://learn.adafruit.com/arduino-clock/overview https://learn.adafruit.com/arduino-clock/overview
erentz 9y ago
Should have a 2012 in it. Can someone clarify something for me. The GPS part gets talked about a lot in these articles, but it's never been clear why. Does Google go as far as putting a GPS receiver in every server? If not, what is so special about this? (People have been using GPS as a time source on networks for decades. But the way articles talk about it, this is super-duper novel.)
- xbmcuser 9y ago
  They have attached atomic clocks to their servers let alone gps
- packetslave 9y ago
  Take a look at Spanner paper for details. There’s a set of time master machines per datacenter that have GPS receivers attached, and a set of “Armageddon masters” that have atomic clocks attached. Regular machines (including the Spanner servers) run a time slave daemon that talks to the master machines.
  - sleavey 9y ago
    Naively I'd assume that the improvement in time accuracy GPS clocks provide must be greater than the harmful effect on accuracy created by network latency, even within data centres: you can't guarantee that a packet will arrive on time, since switches can, I believe as part of their protocol, apply a random wait to a packet. I assume there must be some special method to maintain a synchronous connection to the clock on each server over the network. Could it be as simple as some QoS rules in the switches for NTP packets?
- danbruc 9y ago
  If you want consistency in a distributed system, you have to establish a consistent ordering of events across the system. In absence of synchronized clocks you bounce back and forth logical timestamps, essentially numbers increasing whenever an event occurs because you only really care whether A happend before or after B, not how much time elapsed in between. This adds latency for the additional communication and usually scales not well with the number of processes you have to synchronize. See for example Lamport or vector clocks. With accurate enough clocks you can establish a global event order simply by time stamping events with the actual time. If you want to process events at really high frequency, you need a correspondingly accurate synchronization between the clocks. The idea is decades old, I guess it is more of an engineering challenge. With the number of systems Google operates at least a few things will break all the time and you have to be able to tolerate that. But I think I just skimmed the paper when it came out and might therefore be unaware of some important details.
- InclinedPlane 9y ago
  GPS is "just" each satellite sending out a constant readout of its current time, very precisely. By reading these from several satellites it's possible to figure out what the actual current time is and then from that determine the time of flight of each signal. Which gives you the distance to each satellite, so if you know where each satellite is you can figure out your own position too. The point is, it's time based, and it enables synchronization of clocks across the globe to sub-microsecond accuracy. Google uses that feature for keeping its distributed databases synchronized as well. It's not just an alternative to NTP, it's a way to keep systems in sync to a very, very precise degree. Which requires constant communication with time reference servers. They don't put a GPS receiver in every server, just a redundant cluster of them in each data center. Consider a transactional write using Spanner, the way it works is that the system connects to all of the relevant database shards, it knows what all of their clock accuracy levels are and so it knows how long it needs to wait to negotiate a transaction lock. This is only possible due to the high accuracy of time synchronization. Meanwhile it's buffering write data/instructions to all the hosts. Once it gets the lock it then sends out commit instructions. The whole process can take mere tens of milliseconds for a handful of hosts and only a few hundreds of milliseconds for hundreds of hosts. Which is remarkable when you consider the scale of the problem.
  - erentz 9y ago
    > They don't put a GPS receiver in every server, just a redundant cluster of them in each data center. Ok that’s what I figured. The way the media reported on this the use of GPS as a time source was the special big deal. And I was trying to work out why because there was nothing novel in putting GPS time sources in data centers. (We did that at my first job not quite 2 decades ago...)

36 points by sds111 9y ago

mehrdada 9y ago

(2012)

ars 9y ago

This article explains exactly nothing. You can replace the word TrueTime, with "Magic" and learn exactly as much.

sumitgt 9y ago
The recent paper on Spanner has some details of how it is used if you are interested. https://static.googleusercontent.com/media/research.google.com/en//archive/spanner-osdi2012.pdf https://static.googleusercontent.com/media/research.google.c...

itronitron 9y ago

you can make one of these yourself now if you need highly accurate (but not completely accurate all the time) time stamps... https://learn.adafruit.com/arduino-clock/overview https://learn.adafruit.com/arduino-clock/overview

erentz 9y ago

Should have a 2012 in it. Can someone clarify something for me. The GPS part gets talked about a lot in these articles, but it's never been clear why. Does Google go as far as putting a GPS receiver in every server? If not, what is so special about this? (People have been using GPS as a time source on networks for decades. But the way articles talk about it, this is super-duper novel.)

xbmcuser 9y ago
They have attached atomic clocks to their servers let alone gps
packetslave 9y ago
Take a look at Spanner paper for details. There’s a set of time master machines per datacenter that have GPS receivers attached, and a set of “Armageddon masters” that have atomic clocks attached. Regular machines (including the Spanner servers) run a time slave daemon that talks to the master machines.
- sleavey 9y ago
  Naively I'd assume that the improvement in time accuracy GPS clocks provide must be greater than the harmful effect on accuracy created by network latency, even within data centres: you can't guarantee that a packet will arrive on time, since switches can, I believe as part of their protocol, apply a random wait to a packet. I assume there must be some special method to maintain a synchronous connection to the clock on each server over the network. Could it be as simple as some QoS rules in the switches for NTP packets?
danbruc 9y ago
If you want consistency in a distributed system, you have to establish a consistent ordering of events across the system. In absence of synchronized clocks you bounce back and forth logical timestamps, essentially numbers increasing whenever an event occurs because you only really care whether A happend before or after B, not how much time elapsed in between. This adds latency for the additional communication and usually scales not well with the number of processes you have to synchronize. See for example Lamport or vector clocks. With accurate enough clocks you can establish a global event order simply by time stamping events with the actual time. If you want to process events at really high frequency, you need a correspondingly accurate synchronization between the clocks. The idea is decades old, I guess it is more of an engineering challenge. With the number of systems Google operates at least a few things will break all the time and you have to be able to tolerate that. But I think I just skimmed the paper when it came out and might therefore be unaware of some important details.
InclinedPlane 9y ago
GPS is "just" each satellite sending out a constant readout of its current time, very precisely. By reading these from several satellites it's possible to figure out what the actual current time is and then from that determine the time of flight of each signal. Which gives you the distance to each satellite, so if you know where each satellite is you can figure out your own position too. The point is, it's time based, and it enables synchronization of clocks across the globe to sub-microsecond accuracy. Google uses that feature for keeping its distributed databases synchronized as well. It's not just an alternative to NTP, it's a way to keep systems in sync to a very, very precise degree. Which requires constant communication with time reference servers. They don't put a GPS receiver in every server, just a redundant cluster of them in each data center. Consider a transactional write using Spanner, the way it works is that the system connects to all of the relevant database shards, it knows what all of their clock accuracy levels are and so it knows how long it needs to wait to negotiate a transaction lock. This is only possible due to the high accuracy of time synchronization. Meanwhile it's buffering write data/instructions to all the hosts. Once it gets the lock it then sends out commit instructions. The whole process can take mere tens of milliseconds for a handful of hosts and only a few hundreds of milliseconds for hundreds of hosts. Which is remarkable when you consider the scale of the problem.
- erentz 9y ago
  > They don't put a GPS receiver in every server, just a redundant cluster of them in each data center. Ok that’s what I figured. The way the media reported on this the use of GPS as a time source was the special big deal. And I was trying to work out why because there was nothing novel in putting GPS time sources in data centers. (We did that at my first job not quite 2 decades ago...)