4 ms·

I think it would be super interesting if they started offering IEEE 1588 v2 aka Precision Time Protocol, but it would be much harder to offer compared to ntp. W

by SEJeff 7y ago

I think it would be super interesting if they started offering IEEE 1588 v2 aka Precision Time Protocol, but it would be much harder to offer compared to ntp. What they're almost certainly doing here is just running a cable to the roof where they have a GPS antenna and then run it into their datacenters into a time appliance (less likely) or a GPS pci card in one of their servers (more likely) that they then send out to those who want access.

ixf 7y ago
You really can't do PTP over the internet, at least not meaningfully. Anyone who needs PTP is going to have their own grandmasters, reference sources, etc - and then a distribution network. Most telecoms applications use an ePRTC source which tends to be implemented as a GPS/GLONASS/Galileo redundant frequency source, plus a local rubidium source or cesium reference. High-end telecoms applications use a hydrogen maser. You can't stuff that over an unmanaged network and get the performance you need. Hardware needs to support it hop-to-hop.
- SEJeff 7y ago
  Yeah I agree, but ptpv2 does support unicast and you can set offsets. It would be very challenging, but not impossible sans the very high res stuff where you need oscillators on the switches and routers, as you alluded to. It wouldn't get down to the sub-10 nanosecond sync you get with a proper stratum 1 timesource (such as the rubidium decay ones), but you could get faster than the guaranteed 1 second of accuracy which is what I believe ntp used to guarantee from a protocol level. That was my point. Also if you're inside one of Cloudflare's many POPs, this could, in theory, be provided.
  - mlichvar 7y ago
    PTP over Internet doesn't make much sense. PTP requires hardware support in all network devices on the path between the (grand)master and slave. Without this support it will generally perform worse than NTP. Of course, it depends also on the implementation. PTP does support unicast messaging, but it is not meant to be used as a public service. There are two major problems: It's not stateless and it has a huge traffic amplification, which could be easily exploited for DoS attacks.
    - SEJeff 7y ago
      Yeah this is the most sensible reason against it. NTP is also responsible for some of the biggest traffic amplification DDoS events.
      - mlichvar 7y ago
        Yes, but NTP as a time service (client/server mode) is safe. A request has a single response and their lengths are symmetric (that's actually a requirement for accurate synchronization). The problem with amplification is in the optional monitoring/control modes of the protocol (modes 6 and 7 as used by the ntpq and ntpdc utilities respectively), which should be disabled on public servers. Unfortunately, there are still some old misconfigured servers causing problems for a lot of people. In PTP the problem is in the synchronization protocol itself. A master in the unicast mode is basically a programmable packet generator. It sends sync/announce messages at a rate and duration specified by its slaves, and the address can be spoofed.
rkeene2 7y ago
For a project I was working I needed to synchronize clocks with extremely high precision (though accuracy was less important, and it was masterless so there was no authority) similar to PTP and I created the following simple system: 1. Server: Connect over TCP. For every byte sent, responds back with that system's time in microsceonds. 2. Client: Select a few peers, one at a time: connect over TCP, send a byte and note the round-trip time, halve it to get an approximation of the timepoint that the value was sent to you at, compute delta from current system clock; repeat 20 times, discard first 5 values; exclude outliers (+/- 1 standard deviation) and average to compute the offset, then repeat for the next host, etc until you get some change to make It worked out and kept the required precision among the collection of nodes required for Ceph to work well.

by SEJeff 7y ago

ixf 7y ago

You really can't do PTP over the internet, at least not meaningfully. Anyone who needs PTP is going to have their own grandmasters, reference sources, etc - and then a distribution network. Most telecoms applications use an ePRTC source which tends to be implemented as a GPS/GLONASS/Galileo redundant frequency source, plus a local rubidium source or cesium reference. High-end telecoms applications use a hydrogen maser. You can't stuff that over an unmanaged network and get the performance you need. Hardware needs to support it hop-to-hop.

SEJeff 7y ago
Yeah I agree, but ptpv2 does support unicast and you can set offsets. It would be very challenging, but not impossible sans the very high res stuff where you need oscillators on the switches and routers, as you alluded to. It wouldn't get down to the sub-10 nanosecond sync you get with a proper stratum 1 timesource (such as the rubidium decay ones), but you could get faster than the guaranteed 1 second of accuracy which is what I believe ntp used to guarantee from a protocol level. That was my point. Also if you're inside one of Cloudflare's many POPs, this could, in theory, be provided.
- mlichvar 7y ago
  PTP over Internet doesn't make much sense. PTP requires hardware support in all network devices on the path between the (grand)master and slave. Without this support it will generally perform worse than NTP. Of course, it depends also on the implementation. PTP does support unicast messaging, but it is not meant to be used as a public service. There are two major problems: It's not stateless and it has a huge traffic amplification, which could be easily exploited for DoS attacks.
  - SEJeff 7y ago
    Yeah this is the most sensible reason against it. NTP is also responsible for some of the biggest traffic amplification DDoS events.
    - mlichvar 7y ago
      Yes, but NTP as a time service (client/server mode) is safe. A request has a single response and their lengths are symmetric (that's actually a requirement for accurate synchronization). The problem with amplification is in the optional monitoring/control modes of the protocol (modes 6 and 7 as used by the ntpq and ntpdc utilities respectively), which should be disabled on public servers. Unfortunately, there are still some old misconfigured servers causing problems for a lot of people. In PTP the problem is in the synchronization protocol itself. A master in the unicast mode is basically a programmable packet generator. It sends sync/announce messages at a rate and duration specified by its slaves, and the address can be spoofed.

rkeene2 7y ago

For a project I was working I needed to synchronize clocks with extremely high precision (though accuracy was less important, and it was masterless so there was no authority) similar to PTP and I created the following simple system: 1. Server: Connect over TCP. For every byte sent, responds back with that system's time in microsceonds. 2. Client: Select a few peers, one at a time: connect over TCP, send a byte and note the round-trip time, halve it to get an approximation of the timepoint that the value was sent to you at, compute delta from current system clock; repeat 20 times, discard first 5 values; exclude outliers (+/- 1 standard deviation) and average to compute the offset, then repeat for the next host, etc until you get some change to make It worked out and kept the required precision among the collection of nodes required for Ceph to work well.