3 ms·

Like Spanner, Cockroach’s correctness depends on the strength of its clocks... Unlike Spanner, CockroachDB users are likely deploying on commodity hardware or t

Like Spanner, Cockroach’s correctness depends on the strength of its clocks... Unlike Spanner, CockroachDB users are likely deploying on commodity hardware or the cloud, without GPS and atomic clocks for reference. I'm curious, do any of the cloud providers offer instances with high-precision clocks? Maybe this is something one of the smaller cloud providers might want to offer to differentiate themselves - instances with GSP clocks.

Already__Taken 10y ago
Why can't these datacenters be using a high precision gps source for their NTP source in the building?
- tdrd 10y ago
  [Cockroach Labs employee] In addition to being extremely precise, TrueTime is explicit about error in its measurements. In other words, a call to TrueTime returns a lower and upper bound for the current time, which is important for systems like Spanner or CockroachDB when operations must be linearized. In effect, Spanner sleeps on each operation for a duration of [latest_upper_bound - current_lower_bound] to ensure that operations are strictly ordered. Your suggestion might improve typical clock synchronization, but still leaves defining the critical offset threshold to the operator, and importantly leaves little choice for software like CockroachDB other than to crash when it detects that the invariant is violated.
- shin_lao 10y ago
  NTP isn't enough, you need PTP.
mikecb 10y ago
The spanner announcement mentioned offering direct access to the TrueTime api sometime in the not too distant future. Edit: Quote from Brewer's post: "Next steps For a significantly deeper dive into the details, see the white paper also released today. It covers Spanner, consistency and availability in depth (including new data). It also looks at the role played by Google’s TrueTime system, which provides a globally synchronized clock. We intend to release TrueTime for direct use by Cloud customers in the future."
- aphyr 10y ago
  Bingo. I've been hoping Google (or, heck, any colo provider) would offer this for a few years. Semi-synchronous clocks enable significant performance optimizations!
BrentOzar 10y ago
> I'm curious, do any of the cloud providers offer instances with high-precision clocks? Essentially that's what Google's doing by selling Spanner, but you don't get access to the clocks directly. I'd guess that as they get more experience selling Spanner, and they bring the price of the hardware down, they'll rent those directly too.
- nindalf 10y ago
  They're planning to offer direct access to the TrueTime API soon according to the post by Eric Brewer.
tyingq 10y ago
>I'm curious, do any of the cloud providers offer instances with high-precision clocks? That's a great question. Could be a little tricky with AWS for example. It's typical to run a database on a VPC (virtual private cloud), since that layer of the stack doesn't need to be exposed to the Internet. Unfortunately, that means the servers in the VPC can't get to the ⁎.amazon.pool.ntp.org servers. Skimming around, I don't see any of the cloud providers, VPS providers, or dedicated hosting companies offering anything special, like a local stratum 1 time source for each region/location.

Already__Taken 10y ago

Why can't these datacenters be using a high precision gps source for their NTP source in the building?

tdrd 10y ago
[Cockroach Labs employee] In addition to being extremely precise, TrueTime is explicit about error in its measurements. In other words, a call to TrueTime returns a lower and upper bound for the current time, which is important for systems like Spanner or CockroachDB when operations must be linearized. In effect, Spanner sleeps on each operation for a duration of [latest_upper_bound - current_lower_bound] to ensure that operations are strictly ordered. Your suggestion might improve typical clock synchronization, but still leaves defining the critical offset threshold to the operator, and importantly leaves little choice for software like CockroachDB other than to crash when it detects that the invariant is violated.
shin_lao 10y ago
NTP isn't enough, you need PTP.

mikecb 10y ago

The spanner announcement mentioned offering direct access to the TrueTime api sometime in the not too distant future. Edit: Quote from Brewer's post: "Next steps For a significantly deeper dive into the details, see the white paper also released today. It covers Spanner, consistency and availability in depth (including new data). It also looks at the role played by Google’s TrueTime system, which provides a globally synchronized clock. We intend to release TrueTime for direct use by Cloud customers in the future."

aphyr 10y ago
Bingo. I've been hoping Google (or, heck, any colo provider) would offer this for a few years. Semi-synchronous clocks enable significant performance optimizations!

BrentOzar 10y ago

> I'm curious, do any of the cloud providers offer instances with high-precision clocks? Essentially that's what Google's doing by selling Spanner, but you don't get access to the clocks directly. I'd guess that as they get more experience selling Spanner, and they bring the price of the hardware down, they'll rent those directly too.

nindalf 10y ago
They're planning to offer direct access to the TrueTime API soon according to the post by Eric Brewer.

tyingq 10y ago

>I'm curious, do any of the cloud providers offer instances with high-precision clocks? That's a great question. Could be a little tricky with AWS for example. It's typical to run a database on a VPC (virtual private cloud), since that layer of the stack doesn't need to be exposed to the Internet. Unfortunately, that means the servers in the VPC can't get to the ⁎.amazon.pool.ntp.org servers. Skimming around, I don't see any of the cloud providers, VPS providers, or dedicated hosting companies offering anything special, like a local stratum 1 time source for each region/location.