4 ms·

> When a node exceeds the clock offset threshold, it will automatically shut down to prevent anomalies. If you're planning to run on VMware, be prepared to han

> When a node exceeds the clock offset threshold, it will automatically shut down to prevent anomalies. If you're planning to run on VMware, be prepared to handle rather dramatic system clock shifts. I've seen shifts of up to 5 minutes during heavy backup windows. Not all customers might be willing to have their nodes go down due to system clock / NTP issues.

tschottdorf 9y ago
employee here. Yep, we've also had our share of troubles with noisy clock on cloud environments, so that's something we're very aware of. Further down the road, we're considering a "clockless" mode, which of course isn't clockless, but depends less on the offset threshold: https://github.com/cockroachdb/cockroach/issues/14093 https://github.com/cockroachdb/cockroach/issues/14093 That said, even today, configuring a cluster with a fairly high maximum clock offset is feasible for many workloads.
newman314 9y ago
Do people not run NTP on their VMs? Or are you saying that you see heavy clock skew despite having NTP in place?
- arjunnarayan 9y ago
  The latter. NTP only checks and corrects clock offsets every so often. If the "hardware"[1] clock undergoes offset shifts at random times because of VM pauses this won't get fixed immediately until the next NTP sync. This gets exacerbated in cloud settings where VMs get moved between physical machines, or racks since now it's not just the pause, its that the clock is now pointing to a new hardware time source. [1] in quotes since it's viewed as a single piece of hardware to the software inside the VM.
- deleted 9y ago
  [deleted]
- nvarsj 9y ago
  Cassandra user here in AWS. Clock drift is a big problem on VMs. NTP is not aggressive enough in these environments to keep clocks relatively in sync. We regularly had several hundred milli drifts between nodes. As cassandra is extremely clock sensitive, this is a big problem. We ended up using chrony with very aggressive settings to keep things in the sub-ms range for the most part. But it's still possible to get "hiccups" where time will skip. Especially if you reboot a VM. Vanilla ntp makes assumptions about the hardware clock (that drift is stable) that don't apply to virtualised clocks. Using tsc clocksource may help as well.
  - newman314 9y ago
    Interesting. I wonder if anyone has documented any best practices for timekeeping in VMs. VMware has this but it does not appear to have been updated in a while. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427 https://kb.vmware.com/selfservice/microsites/search.do?langu...
    - andy_ppp 9y ago
      Why does VM ware emulate the hardware clock rather than giving (possibly slightly debounced) access to the real system clock?
      - koffiezet 9y ago
        I suppose for vMotion purposes. A VM is not tied to a physical machine.
    - gerdesj 9y ago
      I run a lot of VMware: * Set the esxis to have five external sources * Search fwenable-ntpd (https://www.v-front.de/2012/01/howto-use-esxi-5-as-ntp-server-or-howto.html https://www.v-front.de/2012/01/howto-use-esxi-5-as-ntp-serve...) and download the .vib (do a security audit on it - its a zip file I think - to ensure it is what you think it is). Install the .vib which simple adds a ntp daemon option to the firewall ports. This works on v6.5 * Run ntpd on Linux VMs, pointed at the hosts with the local clock fudge as a fallback * For Windows VMs in a domain, set the AD DC with PDC emulator role to sync its clock to the host via the VM guest tools, leave the rest alone * On your monitoring system make sure that it has an independent list of five sources and use plugins like ntp-peer for ntpds and ntp-time for Windows (Nagios/Icinga etc) With the above recipe, ntpq -p <host> shows offsets less than 1 ms across the board for ntpds after stabilising.

tschottdorf 9y ago

employee here. Yep, we've also had our share of troubles with noisy clock on cloud environments, so that's something we're very aware of. Further down the road, we're considering a "clockless" mode, which of course isn't clockless, but depends less on the offset threshold: https://github.com/cockroachdb/cockroach/issues/14093 https://github.com/cockroachdb/cockroach/issues/14093 That said, even today, configuring a cluster with a fairly high maximum clock offset is feasible for many workloads.

newman314 9y ago

Do people not run NTP on their VMs? Or are you saying that you see heavy clock skew despite having NTP in place?

arjunnarayan 9y ago
The latter. NTP only checks and corrects clock offsets every so often. If the "hardware"[1] clock undergoes offset shifts at random times because of VM pauses this won't get fixed immediately until the next NTP sync. This gets exacerbated in cloud settings where VMs get moved between physical machines, or racks since now it's not just the pause, its that the clock is now pointing to a new hardware time source. [1] in quotes since it's viewed as a single piece of hardware to the software inside the VM.
deleted 9y ago
[deleted]
nvarsj 9y ago
Cassandra user here in AWS. Clock drift is a big problem on VMs. NTP is not aggressive enough in these environments to keep clocks relatively in sync. We regularly had several hundred milli drifts between nodes. As cassandra is extremely clock sensitive, this is a big problem. We ended up using chrony with very aggressive settings to keep things in the sub-ms range for the most part. But it's still possible to get "hiccups" where time will skip. Especially if you reboot a VM. Vanilla ntp makes assumptions about the hardware clock (that drift is stable) that don't apply to virtualised clocks. Using tsc clocksource may help as well.
- newman314 9y ago
  Interesting. I wonder if anyone has documented any best practices for timekeeping in VMs. VMware has this but it does not appear to have been updated in a while. https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427 https://kb.vmware.com/selfservice/microsites/search.do?langu...
  - andy_ppp 9y ago
    Why does VM ware emulate the hardware clock rather than giving (possibly slightly debounced) access to the real system clock?
    - koffiezet 9y ago
      I suppose for vMotion purposes. A VM is not tied to a physical machine.
  - gerdesj 9y ago
    I run a lot of VMware: * Set the esxis to have five external sources * Search fwenable-ntpd (https://www.v-front.de/2012/01/howto-use-esxi-5-as-ntp-server-or-howto.html https://www.v-front.de/2012/01/howto-use-esxi-5-as-ntp-serve...) and download the .vib (do a security audit on it - its a zip file I think - to ensure it is what you think it is). Install the .vib which simple adds a ntp daemon option to the firewall ports. This works on v6.5 * Run ntpd on Linux VMs, pointed at the hosts with the local clock fudge as a fallback * For Windows VMs in a domain, set the AD DC with PDC emulator role to sync its clock to the host via the VM guest tools, leave the rest alone * On your monitoring system make sure that it has an independent list of five sources and use plugins like ntp-peer for ntpds and ntp-time for Windows (Nagios/Icinga etc) With the above recipe, ntpq -p <host> shows offsets less than 1 ms across the board for ntpds after stabilising.