5 ms·

part 2: https://blog.pivotal.io/labs/labs/ntp-server-costing-500year-part-2-characterizing-ntp-clients https://blog.pivotal.io/labs/labs/ntp-server-costing-500y

by sigio 10y ago

part 2: https://blog.pivotal.io/labs/labs/ntp-server-costing-500year-part-2-characterizing-ntp-clients https://blog.pivotal.io/labs/labs/ntp-server-costing-500year...

rincebrain 10y ago
I wish he'd explained somewhere how they leapt to examining virtualized NTP clients, or what they ultimately did (since there's no part 3 that I can find).
- wattt 10y ago
  I too was expecting some sort of conclusion.
  - brian_cunnie 10y ago
    [author] My bad. Now that the post has been picked up on HN, I'll try to write up a conclusion.
- spydum 10y ago
  Virtualization and time sync have had notorious problems. One ugly work around was frequent NTP polling and adjustments. NTP has a min and max poll interval, and it determines how frequently it should poll automatically based on how far it sees drift happening. If it drifts pretty fast, it will quickly gravitate to the minpoll value, which is exactly what they show in their first graph: tons of polling at the minfrequency for certain hypervisors.
  - raverbashing 10y ago
    I wonder, should this be solved with local NTP servers that resolve from the name pool.ntp.org?
    - rhizome 10y ago
      I've found it to be good practice to run a single/few stratum 2 node(s) to serve local resources. To whatever degree, it's usually more important that these resources be more in sync with each other than with a satellite, which is fostered by having as few nodes as possible trying their luck over the internet to bogged down public stratum 1 NTP sources, instead configuring them to use a single source of time from a box more in its vicinity.
    - snuxoll 10y ago
      Most corporate environments have long defaulted to having a strata 2 NTP server, as long as they use Windows since Kerberos requires clocks to be roughly in sync. With the advent of the cloud I see a lot of businesses forgoing any central IdM solution like AD or IPA and as a result don't tend to have an NTP server unless they explicitly configured one. All of my servers are joined to an AD or IPA domain, so they all use my local NTP servers by default.
  - rincebrain 10y ago
    I knew that was once an issue, I'm surprised to find it's still an issue these days. In addition, even with minpoll, that's still a hell of a lot of clients abruptly polling him abusively.
- brian_cunnie 10y ago
  [author] > I wish he'd explained somewhere how they leapt to examining virtualized NTP clients... I had a hunch [wrongly] that the traffic was caused by a particular operating system. I didn't have enough machines to run the tests on bare-metal, so I virtualized them. And I suspected that virtualization would provide a worst-case scenario (the virtualized clocks would be jittery). My big surprise was that Windows was a model client (once per day), OS X was good, too, and that FreeBSD was the worst and Ubuntu a close second. It was a complete inversion of what I had expected to find. > what they ultimately did (since there's no part 3 that I can find). I never wrote part 3, but now that it seems the post has gained traction on Hacker News I might be inspired to write one. The summary would be as follows: To reduce your costs by a quarter, use the following lines in your NTP configuration file to throttle overly-aggressive clients (the important directive is `limited`: ``` restrict default limited kod nomodify notrap nopeer discard minimum 0 ``` Here is a description of NTP rate-limiting and why `limited`, `kod`, and `discard` are important: https://www.eecis.udel.edu/~mills/ntp/html/rate.html https://www.eecis.udel.edu/~mills/ntp/html/rate.html Here is my current NTP configuration: https://github.com/cunnie/deployments/blob/95e9c71e882d453ec0fd168f3a8030ea0707c05a/concourse-ntp-pdns-gce.yml#L178-L192 https://github.com/cunnie/deployments/blob/95e9c71e882d453ec...
  - dfc 10y ago
    I have never understood why people think kod is a useful setting. Why do you think a misbehaving/improperly configured client is going to honor the kod packet? The kod packet helps with some clients but I have never seen it change the behavior of the most egregious abusers. Just ignore future requests from misbehaving clients, there is not a lot of benefit in saying "please stop misbehaving" to a client that does not follow the spec. Your configuration is lacking a number two big best practices. The most glaring is that you really need to add 'iburst' to your server stanzas. After that you should think about adjust minsane and minclock.
    - AstroJetson 10y ago
      Can you post a few more lines about this? I'm in the process of standing up a server and would like to know more about controlling the load. Thanks!

by sigio 10y ago

part 2: https://blog.pivotal.io/labs/labs/ntp-server-costing-500year-part-2-characterizing-ntp-clients https://blog.pivotal.io/labs/labs/ntp-server-costing-500year...

rincebrain 10y ago

I wish he'd explained somewhere how they leapt to examining virtualized NTP clients, or what they ultimately did (since there's no part 3 that I can find).

wattt 10y ago
I too was expecting some sort of conclusion.
- brian_cunnie 10y ago
  [author] My bad. Now that the post has been picked up on HN, I'll try to write up a conclusion.
spydum 10y ago
Virtualization and time sync have had notorious problems. One ugly work around was frequent NTP polling and adjustments. NTP has a min and max poll interval, and it determines how frequently it should poll automatically based on how far it sees drift happening. If it drifts pretty fast, it will quickly gravitate to the minpoll value, which is exactly what they show in their first graph: tons of polling at the minfrequency for certain hypervisors.
- raverbashing 10y ago
  I wonder, should this be solved with local NTP servers that resolve from the name pool.ntp.org?
  - rhizome 10y ago
    I've found it to be good practice to run a single/few stratum 2 node(s) to serve local resources. To whatever degree, it's usually more important that these resources be more in sync with each other than with a satellite, which is fostered by having as few nodes as possible trying their luck over the internet to bogged down public stratum 1 NTP sources, instead configuring them to use a single source of time from a box more in its vicinity.
  - snuxoll 10y ago
    Most corporate environments have long defaulted to having a strata 2 NTP server, as long as they use Windows since Kerberos requires clocks to be roughly in sync. With the advent of the cloud I see a lot of businesses forgoing any central IdM solution like AD or IPA and as a result don't tend to have an NTP server unless they explicitly configured one. All of my servers are joined to an AD or IPA domain, so they all use my local NTP servers by default.
- rincebrain 10y ago
  I knew that was once an issue, I'm surprised to find it's still an issue these days. In addition, even with minpoll, that's still a hell of a lot of clients abruptly polling him abusively.
brian_cunnie 10y ago
[author] > I wish he'd explained somewhere how they leapt to examining virtualized NTP clients... I had a hunch [wrongly] that the traffic was caused by a particular operating system. I didn't have enough machines to run the tests on bare-metal, so I virtualized them. And I suspected that virtualization would provide a worst-case scenario (the virtualized clocks would be jittery). My big surprise was that Windows was a model client (once per day), OS X was good, too, and that FreeBSD was the worst and Ubuntu a close second. It was a complete inversion of what I had expected to find. > what they ultimately did (since there's no part 3 that I can find). I never wrote part 3, but now that it seems the post has gained traction on Hacker News I might be inspired to write one. The summary would be as follows: To reduce your costs by a quarter, use the following lines in your NTP configuration file to throttle overly-aggressive clients (the important directive is `limited`: ``` restrict default limited kod nomodify notrap nopeer discard minimum 0 ``` Here is a description of NTP rate-limiting and why `limited`, `kod`, and `discard` are important: https://www.eecis.udel.edu/~mills/ntp/html/rate.html https://www.eecis.udel.edu/~mills/ntp/html/rate.html Here is my current NTP configuration: https://github.com/cunnie/deployments/blob/95e9c71e882d453ec0fd168f3a8030ea0707c05a/concourse-ntp-pdns-gce.yml#L178-L192 https://github.com/cunnie/deployments/blob/95e9c71e882d453ec...
- dfc 10y ago
  I have never understood why people think kod is a useful setting. Why do you think a misbehaving/improperly configured client is going to honor the kod packet? The kod packet helps with some clients but I have never seen it change the behavior of the most egregious abusers. Just ignore future requests from misbehaving clients, there is not a lot of benefit in saying "please stop misbehaving" to a client that does not follow the spec. Your configuration is lacking a number two big best practices. The most glaring is that you really need to add 'iburst' to your server stanzas. After that you should think about adjust minsane and minclock.
  - AstroJetson 10y ago
    Can you post a few more lines about this? I'm in the process of standing up a server and would like to know more about controlling the load. Thanks!