4 ms·

Time synchronization over a network with unknown latency characteristics is fundamentally impossible.

withinboredom 4y ago
Wow. It’s a wonder anyone has a correct time then. /s Sarcasm aside, their NTP servers are less than 2 hops from my router. I know that because of excellent tools like ping and trace route.
- rezonant 4y ago
  Based on your other post, it sounds like you are doing NTP with your ISP and then using PTP to sync your servers on the local network. Nothing about this implies that you are accurate to the global clock within a microsecond, let alone accurate within any number of nanoseconds. With a correctly equipped and configured PTP installation locally, you can expect your clocks to be synchronized on the order of a very small number of microseconds with respect to each other, but the relationship between those and the atomic global clock is something else. If you are using satellite/GPS time sources, then that's another matter. Why then is it relevant to have an NTP hookup at your ISP?
  - withinboredom 4y ago
    With A ISP, not my ISP. Also, I’m syncing with a Stratum 0 servers which are connected to an atomic clock and several GPS clocks. Also, consistency and accuracy are orthogonal to each other. My servers clocks are consistent and fairly accurate in regards to global time.
    - rezonant 4y ago
      Ah yes, I figured as much, this is why I skipped over the GPS and satellite parts initially. Having a high quality highly local NTP source is better than most sources, but it is nowhere near as close as having your own local time sources, which is what a lot of time sensitive installations do.
- protocolture 4y ago
  no-propagate-ttl really does clean up an ISP network.
ikiris 4y ago
ptp to the ToR isn't rocket science. If you need something hyper precise and are a large enough customer, you can get just about anything you need built in even the big clouds. crazy customer needs is what made up most projects I saw or worked with.
atoav 4y ago
Really? It is now Tuesday in central europe and I guess most machines¹ managed to figure out that fact. Most machines will also know it is 8:11. The question is how much resolution do you need and are your clocks accuratly synced enough for the thing you plan to do. You are aware that many of the physics experiments that operate at the edge of what is possible use extremely accurate clocks synced over national and sometimes continental borders? ¹ let's ignore the ones that have been configured incorrectly
- rezonant 4y ago
  Perfect time synchronization over a network with unknown latency is provably impossible. Fairly certain distributed physics experiments will be using atomic clocks that are not synchronized over the network. And/or they'll use direct satellite or GPS time sources which have predictable latencies.
  - t0b1 4y ago
    CERN at least seems to be using a solution that is running over Ethernet[1] but with custom hardware that is probably fairly expensive. They use a single time source and then measure the delay between each switch/node. Though, this is limited by needing to be able to run a cable between each node so idk how your definition of a distributed experiment fits. [1]: http://white-rabbit.web.cern.ch/ http://white-rabbit.web.cern.ch/
- sethammons 4y ago
  We've several thousands of nodes in our system. Our ops people have witnessed clock skew from seconds to months that are transient. Time is constantly synced via ntpd. You can't trust system clocks in a distributed system to ensure ordering. Some reading: https://codeburst.io/why-shouldnt-you-trust-system-clocks-72a82a41df93 https://codeburst.io/why-shouldnt-you-trust-system-clocks-72...
dilyevsky 4y ago
Drop the “over a network with unknown latency characteristics” - it’s cleaner
- bjornsing 4y ago
  Cleaner perhaps, but wrong. If you know that latency is statistically symmetrical (same in both directions, on average) then you can synchronize clocks to arbitrary precision (asymptotically at least).
  - dilyevsky 4y ago
    > If you know that latency is statistically symmetrical (same in both directions, on average) My understanding of relativity is that this is principally unknowable and we only assume that speed of light is same in every direction by convention (every few years there’s a paper that claims they managed to measure one way sol but later it turns out they actually measured two-way in a roundabout way) so you can’t really know that?

Wow. It’s a wonder anyone has a correct time then. /s Sarcasm aside, their NTP servers are less than 2 hops from my router. I know that because of excellent tools like ping and trace route.

rezonant 4y ago
Based on your other post, it sounds like you are doing NTP with your ISP and then using PTP to sync your servers on the local network. Nothing about this implies that you are accurate to the global clock within a microsecond, let alone accurate within any number of nanoseconds. With a correctly equipped and configured PTP installation locally, you can expect your clocks to be synchronized on the order of a very small number of microseconds with respect to each other, but the relationship between those and the atomic global clock is something else. If you are using satellite/GPS time sources, then that's another matter. Why then is it relevant to have an NTP hookup at your ISP?
- withinboredom 4y ago
  With A ISP, not my ISP. Also, I’m syncing with a Stratum 0 servers which are connected to an atomic clock and several GPS clocks. Also, consistency and accuracy are orthogonal to each other. My servers clocks are consistent and fairly accurate in regards to global time.
  - rezonant 4y ago
    Ah yes, I figured as much, this is why I skipped over the GPS and satellite parts initially. Having a high quality highly local NTP source is better than most sources, but it is nowhere near as close as having your own local time sources, which is what a lot of time sensitive installations do.
protocolture 4y ago
no-propagate-ttl really does clean up an ISP network.

ikiris 4y ago

ptp to the ToR isn't rocket science. If you need something hyper precise and are a large enough customer, you can get just about anything you need built in even the big clouds. crazy customer needs is what made up most projects I saw or worked with.

atoav 4y ago

Really? It is now Tuesday in central europe and I guess most machines¹ managed to figure out that fact. Most machines will also know it is 8:11. The question is how much resolution do you need and are your clocks accuratly synced enough for the thing you plan to do. You are aware that many of the physics experiments that operate at the edge of what is possible use extremely accurate clocks synced over national and sometimes continental borders? ¹ let's ignore the ones that have been configured incorrectly

rezonant 4y ago
Perfect time synchronization over a network with unknown latency is provably impossible. Fairly certain distributed physics experiments will be using atomic clocks that are not synchronized over the network. And/or they'll use direct satellite or GPS time sources which have predictable latencies.
- t0b1 4y ago
  CERN at least seems to be using a solution that is running over Ethernet[1] but with custom hardware that is probably fairly expensive. They use a single time source and then measure the delay between each switch/node. Though, this is limited by needing to be able to run a cable between each node so idk how your definition of a distributed experiment fits. [1]: http://white-rabbit.web.cern.ch/ http://white-rabbit.web.cern.ch/
sethammons 4y ago
We've several thousands of nodes in our system. Our ops people have witnessed clock skew from seconds to months that are transient. Time is constantly synced via ntpd. You can't trust system clocks in a distributed system to ensure ordering. Some reading: https://codeburst.io/why-shouldnt-you-trust-system-clocks-72a82a41df93 https://codeburst.io/why-shouldnt-you-trust-system-clocks-72...

dilyevsky 4y ago

Drop the “over a network with unknown latency characteristics” - it’s cleaner

bjornsing 4y ago
Cleaner perhaps, but wrong. If you know that latency is statistically symmetrical (same in both directions, on average) then you can synchronize clocks to arbitrary precision (asymptotically at least).
- dilyevsky 4y ago
  > If you know that latency is statistically symmetrical (same in both directions, on average) My understanding of relativity is that this is principally unknowable and we only assume that speed of light is same in every direction by convention (every few years there’s a paper that claims they managed to measure one way sol but later it turns out they actually measured two-way in a roundabout way) so you can’t really know that?