3 ms·

I had to chuckle when I read this: "As Fikes points out, Google had to install GPS antennas on the roofs of its data centers and connect them to the hardware b

I had to chuckle when I read this: "As Fikes points out, Google had to install GPS antennas on the roofs of its data centers and connect them to the hardware below." This is usually one of the first things an enterprising sysadmin does at companies when they first start thinking about time - drop a GPS receiver on the roof (and they usually come up with a bunch of cool graphs showing where all the satellites are over time). Soon thereafter, and a bit of reading about the NTP protocol, they realize that just adding: server 0.pool.ntp.org server 1.pool.ntp.org server 2.pool.ntp.org server 3.pool.ntp.org to their ntp.conf is sufficient for 99.99% of all endeavors which require accurate time, outside of big physics, and, apparently Google's Spanner Database. This part was a bit incomplete: "Typically, data-center operators keep their servers in sync using what’s called the Network Time Protocol, or NTP. This is essentially an online service that connects machines to the official atomic clocks that keep time for organizations across the world. But because it takes time to move information across a network, this method is never completely accurate," Much of the purpose (and math) behind the NTP protocol is to deal with network lag. And it does a pretty good job doing so. Reading about the True Time Api at: http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/ko//archive/spanner-osdi2012.pdf http://static.googleusercontent.com/external_content/untrust... "This implementation keeps uncertainty small (generally less than 10ms) by using multiple modern clock references (GPS and atomic clocks)" So - apparently 10ms is their breakpoint - 10ms is about the limit of what you can expect out of NTP, so I guess it makes sense that if Google needs to do 10ms or better, something of their own invention would be required. Cool graph on the paper showing that 99.9% of variance across data centers thousands of kilometers apart are < 10ms deviation.

emmelaich 14y ago
Using NTP is ok until your network connection is broken for hours. Think backhoe into the fibre.
- ghshephard 14y ago
  Data Centers don't lose network connectivity - designed to have ingress and egress far enough apart that a local disaster can't take down the entire ring. Also, as I noted, one of the first things that enterprising sysadmins do is toss a Stratum-0 receiver (GPS receiver) on the roof, which feeds into the NTP protocol. Finally - the entire purpose behind TrueTimeAPI is to sync up spanner's commits so they are consistent across the replicated/sharded nodes. Without Network, that database capability would come to a halt way before network time became a problem. The bigger issue is the <10ms requirements. NTP over the Internet does not get you that consistently, and certainly not at the 99.9% success that Google was able to achieve with TrueTime API.

emmelaich 14y ago

Using NTP is ok until your network connection is broken for hours. Think backhoe into the fibre.

ghshephard 14y ago
Data Centers don't lose network connectivity - designed to have ingress and egress far enough apart that a local disaster can't take down the entire ring. Also, as I noted, one of the first things that enterprising sysadmins do is toss a Stratum-0 receiver (GPS receiver) on the roof, which feeds into the NTP protocol. Finally - the entire purpose behind TrueTimeAPI is to sync up spanner's commits so they are consistent across the replicated/sharded nodes. Without Network, that database capability would come to a halt way before network time became a problem. The bigger issue is the <10ms requirements. NTP over the Internet does not get you that consistently, and certainly not at the 99.9% success that Google was able to achieve with TrueTime API.