4 ms·

I wanted to try it before, but the clock synchronicity thing scares me too much to dare. How do you even keep them in line? What happens if I'm monitoring and s

I wanted to try it before, but the clock synchronicity thing scares me too much to dare. How do you even keep them in line? What happens if I'm monitoring and see an elevated clock-skew? Can I even do anything or do things just start to fail mysteriously? I think CockroachDB will be amazing when cloud providers start offering APIs for High-Resolution Clocks as a service, of even better, if I got to opt-in to the system clock being synced to highres, much like in GCP you can upgrade your instance's network to premium tier on the fly.

thanatos_dem 7y ago
NTP is plenty accurate for Cockroach to work correctly. > What happens if I'm monitoring and see an elevated clock-skew? If you have monitoring that can detect that, NTP should also be able to run and correct it, apart from some very weird failure modes of the network itself (like traffic dropping in one direction). > I think CockroachDB will be amazing when cloud providers start offering APIs for High-Resolution Clocks as a service With the hybrid logical clocks that cockroach uses (basically wall time + vector clock), they don’t need to be all that precise. The trade off it makes is giving up linearizability (aka external serializability). Spanner gets that by making every transaction wait for the maximum clock skew (7ms, with their atomic and GPS clock systems). But in order to run on commodity hardware Cockroach forewent it, in favor of allowing more acceptable skew.
- luizfelberti 7y ago
  I'll read up on that, thanks for pointing these things out. One question though, doesn't CockroachDB also have a holdout period of 250ms (default) when reading across shards, kind of similar to Spanner's 7ms holdout on writes? I remember reading something vaguely similar to that somewhere.
  - thanatos_dem 7y ago
    As far as I’m aware there is no explicit holdout period on reads, however, on read you do need quorum of the range holders for consistency sake. The exception here is if you’re talking to the “leaseholder” of that data, in which case it can respond to you directly. If you have a workflow that consistently exercises the same ranges, leaseholders will be moved to the nodes that you’re connected to, in a process called “follow the workload”. If you run a load generator against a cluster you’ll see that - for the first 30-90 seconds or so it’ll have some latency, but after some period of time, the leases should be moved closer to your generator and you should see latencies drop. When you’re looking at data access patterns and partitioning you should keep this in mind, as it can be a big performance improvement if properly considered.
manigandham 7y ago
NTP works fine. This is a very minor risk. VMs on cloud providers are already preconfigured to use NTP with their own time servers so they are very reliable [1][2] and you can always add your own monitoring [3]. Also CRDB is tested with clock-drift and you can read the jepsen report [4]. 1. https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/ https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-ti... 2. https://developers.google.com/time/ https://developers.google.com/time/ 3. https://docs.datadoghq.com/integrations/ntp/ https://docs.datadoghq.com/integrations/ntp/ 4. https://jepsen.io/analyses/cockroachdb-beta-20160829 https://jepsen.io/analyses/cockroachdb-beta-20160829
- Havoc 7y ago
  >NTP works fine. This is a very minor risk. Until it doesn't. Low probability large impact doesn't directly translate to minor risk imo
  - craftinator 7y ago
    I'm terrified of meteors too!
- scottlocklin 7y ago
  Great; so have you used CockroachDB in production? I bet you haven't!
  - manigandham 7y ago
    Yes, read my other comment at the top of this page.

thanatos_dem 7y ago

NTP is plenty accurate for Cockroach to work correctly. > What happens if I'm monitoring and see an elevated clock-skew? If you have monitoring that can detect that, NTP should also be able to run and correct it, apart from some very weird failure modes of the network itself (like traffic dropping in one direction). > I think CockroachDB will be amazing when cloud providers start offering APIs for High-Resolution Clocks as a service With the hybrid logical clocks that cockroach uses (basically wall time + vector clock), they don’t need to be all that precise. The trade off it makes is giving up linearizability (aka external serializability). Spanner gets that by making every transaction wait for the maximum clock skew (7ms, with their atomic and GPS clock systems). But in order to run on commodity hardware Cockroach forewent it, in favor of allowing more acceptable skew.

luizfelberti 7y ago
I'll read up on that, thanks for pointing these things out. One question though, doesn't CockroachDB also have a holdout period of 250ms (default) when reading across shards, kind of similar to Spanner's 7ms holdout on writes? I remember reading something vaguely similar to that somewhere.
- thanatos_dem 7y ago
  As far as I’m aware there is no explicit holdout period on reads, however, on read you do need quorum of the range holders for consistency sake. The exception here is if you’re talking to the “leaseholder” of that data, in which case it can respond to you directly. If you have a workflow that consistently exercises the same ranges, leaseholders will be moved to the nodes that you’re connected to, in a process called “follow the workload”. If you run a load generator against a cluster you’ll see that - for the first 30-90 seconds or so it’ll have some latency, but after some period of time, the leases should be moved closer to your generator and you should see latencies drop. When you’re looking at data access patterns and partitioning you should keep this in mind, as it can be a big performance improvement if properly considered.

manigandham 7y ago

NTP works fine. This is a very minor risk. VMs on cloud providers are already preconfigured to use NTP with their own time servers so they are very reliable [1][2] and you can always add your own monitoring [3]. Also CRDB is tested with clock-drift and you can read the jepsen report [4]. 1. https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/ https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-ti... 2. https://developers.google.com/time/ https://developers.google.com/time/ 3. https://docs.datadoghq.com/integrations/ntp/ https://docs.datadoghq.com/integrations/ntp/ 4. https://jepsen.io/analyses/cockroachdb-beta-20160829 https://jepsen.io/analyses/cockroachdb-beta-20160829

Havoc 7y ago
>NTP works fine. This is a very minor risk. Until it doesn't. Low probability large impact doesn't directly translate to minor risk imo
- craftinator 7y ago
  I'm terrified of meteors too!
scottlocklin 7y ago
Great; so have you used CockroachDB in production? I bet you haven't!
- manigandham 7y ago
  Yes, read my other comment at the top of this page.