6 ms·

I personally enjoyed the challenge of setting up PTP at home. Why would a hacker scoff at nanosecond-level timekeeping —- isn’t the entire internet a “telco/en

by jbronn 6y ago

I personally enjoyed the challenge of setting up PTP at home. Why would a hacker scoff at nanosecond-level timekeeping —- isn’t the entire internet a “telco/enterprise” thing?

jstrong 6y ago
did you write about your experience? I am interested in this but have very limited knowledge about it at this point.
- varjag 6y ago
  Install linuxptp on the boxes you designate as grandmaster and client. On your grandmaster clock host: ptp4l -i eth0 -m On the client: ptp4l -i eth0 -s & If you want to sync client system clock and watch the synchronization process: phc2sys -a -r -m
wmf 6y ago
To do PTP "right" requires every switch to support it and a NIC with hardware timestamps. Also, I've seen claims that PTP is no more precise than a good implementation of NTP.
- markfeathers 6y ago
  ptp is <1us synchronization. From my testing NTP is ~20-60us after about 10 minutes of sync, but it intentionally drifts the phase around. On average, NTP is pretty close. If you look at the white rabbit FPGA PTP updates, its in the ns range. Any kind of GPS + most intel nics will get you PTP with an accurate clock. If you didn't need to sync too many devices you could use a single system with a bunch of nics as your "switch".
  - willis936 6y ago
    This post didn’t sound right to me, but I realized that my raspi4 GPS NTP server has been running ntp and not chrony. Chrony is better at modeling non deterministic timing behavior, so I swapped to that. It’s been ten minutes now and chronyc tracking has been marching the offset down. It’s sub 1 us at this point. System time : 0.000000123 seconds fast of NTP time Last offset : +0.000000366 seconds How to get this precise time out of a non deterministic OS? Beats me. Once I figure that out I can finish my clock project. My best lead is to step through the different python timing and scheduler implementations and see which has the lowest jitter relative to the PPS on an oscilloscope.
    - raginalix 6y ago
      I've always had a lot of trouble getting low jitter with python. C is much better.
    - sgtnoodle 6y ago
      Assuming you're using a PPS signal and a kernel driver, presumably there's an interrupt handler or perhaps a capture timer peripheral that is capturing a hardware timer when the PPS edge occurs. It doesn't matter too much when the userspace code gets around to adjusting the hardware timer as long as it can compute the difference between when the PPS edge came in and when it should have come in. The Linux API for fine tuning the system time works in deltas rather than absolute timestamps, so it is once again fairly immune to userspace scheduling jitter. Even good hardware oscillators can have a wide amount of drift, say 50uS per second, but they tend to be stable over several minutes outside of extreme thermal environments. Therefore, it's pretty easy to estimate and compensate for drift using a PPS signal as a reference. Presumably, that compensation is partially what takes a while for the time daemon to converge on. Additionally, the clock sync daemon likely takes a while to converge because it isn't directly controlling the system time. Rather, it is sending hints to the kernel for it to adjust the time. The kernel decides how best to do that, and it does it in a way that attempts to avoid breaking other userspace programs that are running. For example, it tries to keep system time monotonically increasing. This means that there's relatively low gain in the feedback loop, and so it takes a while to cancel out error. It's possible for a userspace program to instead explicitly set system time, but that really isn't intended to be used in Linux unless time is more than 0.5 seconds off. The API call to do that is inherently vulnerable to userspace scheduling jitter, but it's fine since 0.5 seconds is orders of magnitude longer than the expected jitter. You get the system time within the ballpark, and then incrementally adjust it until it's perfect. If you're not using a kernel driver to capture the PPS edge's timestamp, then you're going to have a rougher time. Either you're just going to have to accept the fact that you can't do better than the scheduling jitter (other than assume it averages out), or you're going to have to do something clever/terrible. One idea would be to have your userspace process go to sleep until, say, 1ms before you expect the next PPS edge to come in. Then, go into a tight polling loop until the edge occurs. As long as reading the PPS pin from userspace is non-blocking and your process doesn't get preempted, you should be able to get at least within microseconds. You can poll system time in the same tight loop, allowing you to fairly reliably detect whether the process got preempted or not.
  - varjag 6y ago
    NTP gets worse if you sync more than two devices across a broader network with other switched traffic, more into low 100s of µs. PTP does not degrade similarly and yes, most of PHYs made since middle of the last decade support it.
  - HourglassFR 6y ago
    > If you look at the white rabbit FPGA PTP updates, its in the ns range As I recall, I had even better performance than that. Around the tens of picoseconds. But I guess the advertised 1 ns is a conservative estimate. The precison is incredible but its not magic, they squeeze the maximum amount of determinism out of custom hardware and fiber optic links. It is a bit of pain too set up, as you need to calibrate each link individually every time you change the fiber or the SFP.
- makomk 6y ago
  The NIC with hardware timestamps part should be pretty easy if you're already implementing it using an FPGA like this project did - in fact, in a sense that seems to be exactly what they're doing with NTP. Finding switches that support it might be a little harder.
- Unklejoe 6y ago
  > To do PTP "right" requires every switch to support it and a NIC with hardware timestamps. I agree, but PTP will in fact work over regular commercial switches on a LAN. The problem is that it will introduce jitter if there's other traffic on the network, but as long as the paths from master to slave (terms used in the standard) remain symmetric, you can filter this out and achieve performance almost as good as if you were using PTP transparent switches.
wyldfire 6y ago
Many many computers on the internet have a RTC with second resolution and timer tick resolution in the microseconds. Reference time resolution that's that much higher than your timer tick period is useless.

by jbronn 6y ago

I personally enjoyed the challenge of setting up PTP at home. Why would a hacker scoff at nanosecond-level timekeeping —- isn’t the entire internet a “telco/enterprise” thing?

jstrong 6y ago

did you write about your experience? I am interested in this but have very limited knowledge about it at this point.

varjag 6y ago
Install linuxptp on the boxes you designate as grandmaster and client. On your grandmaster clock host: ptp4l -i eth0 -m On the client: ptp4l -i eth0 -s & If you want to sync client system clock and watch the synchronization process: phc2sys -a -r -m

wmf 6y ago

To do PTP "right" requires every switch to support it and a NIC with hardware timestamps. Also, I've seen claims that PTP is no more precise than a good implementation of NTP.

markfeathers 6y ago
ptp is <1us synchronization. From my testing NTP is ~20-60us after about 10 minutes of sync, but it intentionally drifts the phase around. On average, NTP is pretty close. If you look at the white rabbit FPGA PTP updates, its in the ns range. Any kind of GPS + most intel nics will get you PTP with an accurate clock. If you didn't need to sync too many devices you could use a single system with a bunch of nics as your "switch".
- willis936 6y ago
  This post didn’t sound right to me, but I realized that my raspi4 GPS NTP server has been running ntp and not chrony. Chrony is better at modeling non deterministic timing behavior, so I swapped to that. It’s been ten minutes now and chronyc tracking has been marching the offset down. It’s sub 1 us at this point. System time : 0.000000123 seconds fast of NTP time Last offset : +0.000000366 seconds How to get this precise time out of a non deterministic OS? Beats me. Once I figure that out I can finish my clock project. My best lead is to step through the different python timing and scheduler implementations and see which has the lowest jitter relative to the PPS on an oscilloscope.
  - raginalix 6y ago
    I've always had a lot of trouble getting low jitter with python. C is much better.
  - sgtnoodle 6y ago
    Assuming you're using a PPS signal and a kernel driver, presumably there's an interrupt handler or perhaps a capture timer peripheral that is capturing a hardware timer when the PPS edge occurs. It doesn't matter too much when the userspace code gets around to adjusting the hardware timer as long as it can compute the difference between when the PPS edge came in and when it should have come in. The Linux API for fine tuning the system time works in deltas rather than absolute timestamps, so it is once again fairly immune to userspace scheduling jitter. Even good hardware oscillators can have a wide amount of drift, say 50uS per second, but they tend to be stable over several minutes outside of extreme thermal environments. Therefore, it's pretty easy to estimate and compensate for drift using a PPS signal as a reference. Presumably, that compensation is partially what takes a while for the time daemon to converge on. Additionally, the clock sync daemon likely takes a while to converge because it isn't directly controlling the system time. Rather, it is sending hints to the kernel for it to adjust the time. The kernel decides how best to do that, and it does it in a way that attempts to avoid breaking other userspace programs that are running. For example, it tries to keep system time monotonically increasing. This means that there's relatively low gain in the feedback loop, and so it takes a while to cancel out error. It's possible for a userspace program to instead explicitly set system time, but that really isn't intended to be used in Linux unless time is more than 0.5 seconds off. The API call to do that is inherently vulnerable to userspace scheduling jitter, but it's fine since 0.5 seconds is orders of magnitude longer than the expected jitter. You get the system time within the ballpark, and then incrementally adjust it until it's perfect. If you're not using a kernel driver to capture the PPS edge's timestamp, then you're going to have a rougher time. Either you're just going to have to accept the fact that you can't do better than the scheduling jitter (other than assume it averages out), or you're going to have to do something clever/terrible. One idea would be to have your userspace process go to sleep until, say, 1ms before you expect the next PPS edge to come in. Then, go into a tight polling loop until the edge occurs. As long as reading the PPS pin from userspace is non-blocking and your process doesn't get preempted, you should be able to get at least within microseconds. You can poll system time in the same tight loop, allowing you to fairly reliably detect whether the process got preempted or not.
- varjag 6y ago
  NTP gets worse if you sync more than two devices across a broader network with other switched traffic, more into low 100s of µs. PTP does not degrade similarly and yes, most of PHYs made since middle of the last decade support it.
- HourglassFR 6y ago
  > If you look at the white rabbit FPGA PTP updates, its in the ns range As I recall, I had even better performance than that. Around the tens of picoseconds. But I guess the advertised 1 ns is a conservative estimate. The precison is incredible but its not magic, they squeeze the maximum amount of determinism out of custom hardware and fiber optic links. It is a bit of pain too set up, as you need to calibrate each link individually every time you change the fiber or the SFP.
makomk 6y ago
The NIC with hardware timestamps part should be pretty easy if you're already implementing it using an FPGA like this project did - in fact, in a sense that seems to be exactly what they're doing with NTP. Finding switches that support it might be a little harder.
Unklejoe 6y ago
> To do PTP "right" requires every switch to support it and a NIC with hardware timestamps. I agree, but PTP will in fact work over regular commercial switches on a LAN. The problem is that it will introduce jitter if there's other traffic on the network, but as long as the paths from master to slave (terms used in the standard) remain symmetric, you can filter this out and achieve performance almost as good as if you were using PTP transparent switches.

wyldfire 6y ago

Many many computers on the internet have a RTC with second resolution and timer tick resolution in the microseconds. Reference time resolution that's that much higher than your timer tick period is useless.