5 ms·

The problem with these types of posts is that this is an area that many are unfamiliar with (at least not in depth) and making lots of authoritative statements

The problem with these types of posts is that this is an area that many are unfamiliar with (at least not in depth) and making lots of authoritative statements makes it believable at face value. There are so many variables to network time sync that you design to minimise them. For example no multipath and no asymmetry unless you have PTP P2P transparent clocks everywhere. The author also mixes precision with accuracy and relies on self-reported figures from NTP (chrony says xxx ns jitter). With every media speed change you get asymmetry which affects accuracy (not always precision though). So your 100m->1G link for example will already introduce over 1 us of error (to accuracy!), but NTP will never show you this and nothing will unless you measure both ends with 1PPS, and the only way around it is PTP BC or TC. There is a very long list of similar clarifications that can be made. For example nothing is mentioned about message rates / intervals, which are crucial for improving the numbers of samples filters work with - and the fact that ptp4l and especially phc2sys aren't great with filtering. Finally getting time into the OS clock, unless you use PCIE PTM which practically limits you to newer Intel CPUs and newer Intel NICs, relies on a PCIE transaction with unknown delays and asymmetries, and without PTM (excluding few NICs) your OS clock is nearly always 500+ ns away from the PHC and you don't know by how much and you can't measure it. It's just a complex topic and requires an end to end, leave no stone unturned, semi-scientific approach to really present things correctly.

mlichvar 1y ago
I agree with most of what you said. The author has other posts in the series where he tried to measure the accuracy relative to the PHC (not system clock) using PPS: https://scottstuff.net/posts/2025/06/02/measuring-ntp-accuracy-with-an-oscilloscope/ https://scottstuff.net/posts/2025/06/02/measuring-ntp-accura... Steering the same PHC with phc2sys as chronyd is using for HW timestamping is not the best approach as that creates a feedback loop (instability). It would be better to leave the PHC running free and just compare the sys<->PHC with PHC<->PPS offsets. > So your 100m->1G link for example will already introduce over 1 us of error (to accuracy!), but NTP will never show you this That doesn't apply to NTP to such an extent as PTP because it timestamps end of the reception (HW RX timestamps are transposed to the end of the packet), so the asymmetries in transmission lengths due to different link speeds should cancel out (unless the switches are cutting through in the faster->slower link direction, but that seems to be rare).
- wowczarek 1y ago
  Yes, I saw those PTP posts and where the methodology lacks a bit. Re. asymmetries canceling out, OK, I oversimplified, and this is true in theory and often in practice, but for example, having done this with nearly all generations of enterprise type Broadcom ASICs sort of 2008 onwards I know that there are so many variations to this behaviour that the only way to know is to precisely measure latencies in one direction and the other for a variety of speed, CT vs. S&F and frame sizes, and even bandwidth combinations and see. I used to characterise switches for this, build test harnesses, measurement tools etc., and I saw everything ranging from: CT one way, S&F the other way, but not for all speed combinations, then CT behaviour regardless of enabling or disabling it, finally even things like latency having quantised step characteristics in increments of say X bytes because internally the switching fabric used X-byte cells, and then CT only behaving like CT above certain frame sizes. There's just a lot to take into account. There are even cases where a certain level of background traffic _improves_ latency fairness and symmetry, an equivalent of keeping the caches hot. The author's best bet at reliable numbers would be to get himself a Latte Panda Mu or another platform with TGPIO and measure against 1PPS straight from the CPU. That would be the ultimate answer. Failing that, at least a PTM NIC synced to the OS clock, but that will alter the noise profile of the OS clock. But you and me know all this because we've been digging deep into the hardware and software guts of those things for years, and have done this for a job, and what's a home lab user to do. It's a never-ending learning exercise and the key is to acknowledge the possible unknowns, and by that I don't mean scientific unknowns but that we don't know what we don't know, and bloggers sometimes don't do this.
- eqvinox 1y ago
  > Steering the same PHC with phc2sys as chronyd is using for HW timestamping is not the best approach as that creates a feedback loop (instability). This is standard practice, though, for most PTP slave clocks. The feedback is just factored into the math. (Why? No idea. I just know how the code works.) Although… it's standard practice in PTP setups that are designed for it. Not NTP… if only there were a specification… :) I do have to wonder though. Of what use are timestamps from an unsynchronized PHC to chrony? Is it continuously taking twin sys+PHC timestamps to line up things?
petesoper 1y ago
The OP says early he only needs 10us.
Dylan16807 1y ago
It's not that the author is mixing accuracy and precision, it's that they only care about precision. Any asymmetry that is consistent is irrelevant.
eqvinox 1y ago
ConnectX-6 has PTM, though I have not tested it.
- wowczarek 1y ago
  Do it, that should make a significant difference. See https://www.opencompute.org/wiki/PTM_Readiness https://www.opencompute.org/wiki/PTM_Readiness for other hardware that supports it, i225/226 are the most common these days, but also a system with TGPIO 1PPS will show the real picture.
  - RossBencina 1y ago
    Is the i225/226 datasheet public? I have only ever been able to locate the product brief.
  - eqvinox 1y ago
    > Do it, that should make a significant difference. Need the cards for that first ;). Still on cx5 here. (Also some nVidia docs say you need cx7, but it's listed for cx6, not sure which is true…)
anon6362 1y ago
Sigh Yep. Dunning-Kruger effect specimens hammer out puff pieces to get their participation awards. Meanwhile, here's some other articles: NTP: https://austinsnerdythings.com/2025/02/14/revisiting-microsecond-accurate-ntp-for-raspberry-pi-with-gps-pps-in-2025/ https://austinsnerdythings.com/2025/02/14/revisiting-microse... PTP: https://austinsnerdythings.com/2025/02/18/nanosecond-accurate-ptp-server-grandmaster-and-client-tutorial-for-raspberry-pi/ https://austinsnerdythings.com/2025/02/18/nanosecond-accurat... https://www.jeffgeerling.com/blog/2025/diy-ptp-grandmaster-clock-raspberry-pi https://www.jeffgeerling.com/blog/2025/diy-ptp-grandmaster-c...

mlichvar 1y ago

I agree with most of what you said. The author has other posts in the series where he tried to measure the accuracy relative to the PHC (not system clock) using PPS: https://scottstuff.net/posts/2025/06/02/measuring-ntp-accuracy-with-an-oscilloscope/ https://scottstuff.net/posts/2025/06/02/measuring-ntp-accura... Steering the same PHC with phc2sys as chronyd is using for HW timestamping is not the best approach as that creates a feedback loop (instability). It would be better to leave the PHC running free and just compare the sys<->PHC with PHC<->PPS offsets. > So your 100m->1G link for example will already introduce over 1 us of error (to accuracy!), but NTP will never show you this That doesn't apply to NTP to such an extent as PTP because it timestamps end of the reception (HW RX timestamps are transposed to the end of the packet), so the asymmetries in transmission lengths due to different link speeds should cancel out (unless the switches are cutting through in the faster->slower link direction, but that seems to be rare).

wowczarek 1y ago
Yes, I saw those PTP posts and where the methodology lacks a bit. Re. asymmetries canceling out, OK, I oversimplified, and this is true in theory and often in practice, but for example, having done this with nearly all generations of enterprise type Broadcom ASICs sort of 2008 onwards I know that there are so many variations to this behaviour that the only way to know is to precisely measure latencies in one direction and the other for a variety of speed, CT vs. S&F and frame sizes, and even bandwidth combinations and see. I used to characterise switches for this, build test harnesses, measurement tools etc., and I saw everything ranging from: CT one way, S&F the other way, but not for all speed combinations, then CT behaviour regardless of enabling or disabling it, finally even things like latency having quantised step characteristics in increments of say X bytes because internally the switching fabric used X-byte cells, and then CT only behaving like CT above certain frame sizes. There's just a lot to take into account. There are even cases where a certain level of background traffic _improves_ latency fairness and symmetry, an equivalent of keeping the caches hot. The author's best bet at reliable numbers would be to get himself a Latte Panda Mu or another platform with TGPIO and measure against 1PPS straight from the CPU. That would be the ultimate answer. Failing that, at least a PTM NIC synced to the OS clock, but that will alter the noise profile of the OS clock. But you and me know all this because we've been digging deep into the hardware and software guts of those things for years, and have done this for a job, and what's a home lab user to do. It's a never-ending learning exercise and the key is to acknowledge the possible unknowns, and by that I don't mean scientific unknowns but that we don't know what we don't know, and bloggers sometimes don't do this.
eqvinox 1y ago
> Steering the same PHC with phc2sys as chronyd is using for HW timestamping is not the best approach as that creates a feedback loop (instability). This is standard practice, though, for most PTP slave clocks. The feedback is just factored into the math. (Why? No idea. I just know how the code works.) Although… it's standard practice in PTP setups that are designed for it. Not NTP… if only there were a specification… :) I do have to wonder though. Of what use are timestamps from an unsynchronized PHC to chrony? Is it continuously taking twin sys+PHC timestamps to line up things?

petesoper 1y ago

The OP says early he only needs 10us.

Dylan16807 1y ago

It's not that the author is mixing accuracy and precision, it's that they only care about precision. Any asymmetry that is consistent is irrelevant.

eqvinox 1y ago

ConnectX-6 has PTM, though I have not tested it.

wowczarek 1y ago
Do it, that should make a significant difference. See https://www.opencompute.org/wiki/PTM_Readiness https://www.opencompute.org/wiki/PTM_Readiness for other hardware that supports it, i225/226 are the most common these days, but also a system with TGPIO 1PPS will show the real picture.
- RossBencina 1y ago
  Is the i225/226 datasheet public? I have only ever been able to locate the product brief.
- eqvinox 1y ago
  > Do it, that should make a significant difference. Need the cards for that first ;). Still on cx5 here. (Also some nVidia docs say you need cx7, but it's listed for cx6, not sure which is true…)

anon6362 1y ago

Sigh Yep. Dunning-Kruger effect specimens hammer out puff pieces to get their participation awards. Meanwhile, here's some other articles: NTP: https://austinsnerdythings.com/2025/02/14/revisiting-microsecond-accurate-ntp-for-raspberry-pi-with-gps-pps-in-2025/ https://austinsnerdythings.com/2025/02/14/revisiting-microse... PTP: https://austinsnerdythings.com/2025/02/18/nanosecond-accurate-ptp-server-grandmaster-and-client-tutorial-for-raspberry-pi/ https://austinsnerdythings.com/2025/02/18/nanosecond-accurat... https://www.jeffgeerling.com/blog/2025/diy-ptp-grandmaster-clock-raspberry-pi https://www.jeffgeerling.com/blog/2025/diy-ptp-grandmaster-c...