8 ms·

The last discussion point from this article very much so rings true for me, too. It's been how many years since the Spanner paper and I still can't get a GCP VM

The last discussion point from this article very much so rings true for me, too. It's been how many years since the Spanner paper and I still can't get a GCP VM with atomic clocks? At least now I can provision Cloud Spanner as a managed service, but is this the future of clouds keeping their services at an advantage?

withinboredom 4y ago
Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other. It’s amazing what you can get, if you just ask. It’s the latter part you really want, which is doable with modern hardware accelerated time stamping and a peering NTP system (like chrony).
- bjornsing 4y ago
  Time synchronization over a network with unknown latency characteristics is fundamentally impossible.
  - withinboredom 4y ago
    Wow. It’s a wonder anyone has a correct time then. /s Sarcasm aside, their NTP servers are less than 2 hops from my router. I know that because of excellent tools like ping and trace route.
    - rezonant 4y ago
      Based on your other post, it sounds like you are doing NTP with your ISP and then using PTP to sync your servers on the local network. Nothing about this implies that you are accurate to the global clock within a microsecond, let alone accurate within any number of nanoseconds. With a correctly equipped and configured PTP installation locally, you can expect your clocks to be synchronized on the order of a very small number of microseconds with respect to each other, but the relationship between those and the atomic global clock is something else. If you are using satellite/GPS time sources, then that's another matter. Why then is it relevant to have an NTP hookup at your ISP?
      - withinboredom 4y ago
        With A ISP, not my ISP. Also, I’m syncing with a Stratum 0 servers which are connected to an atomic clock and several GPS clocks. Also, consistency and accuracy are orthogonal to each other. My servers clocks are consistent and fairly accurate in regards to global time.
        rezonant 4y ago
        Ah yes, I figured as much, this is why I skipped over the GPS and satellite parts initially. Having a high quality highly local NTP source is better than most sources, but it is nowhere near as close as having your own local time sources, which is what a lot of time sensitive installations do.
    - protocolture 4y ago
      no-propagate-ttl really does clean up an ISP network.
  - ikiris 4y ago
    ptp to the ToR isn't rocket science. If you need something hyper precise and are a large enough customer, you can get just about anything you need built in even the big clouds. crazy customer needs is what made up most projects I saw or worked with.
  - atoav 4y ago
    Really? It is now Tuesday in central europe and I guess most machines¹ managed to figure out that fact. Most machines will also know it is 8:11. The question is how much resolution do you need and are your clocks accuratly synced enough for the thing you plan to do. You are aware that many of the physics experiments that operate at the edge of what is possible use extremely accurate clocks synced over national and sometimes continental borders? ¹ let's ignore the ones that have been configured incorrectly
    - rezonant 4y ago
      Perfect time synchronization over a network with unknown latency is provably impossible. Fairly certain distributed physics experiments will be using atomic clocks that are not synchronized over the network. And/or they'll use direct satellite or GPS time sources which have predictable latencies.
      - t0b1 4y ago
        CERN at least seems to be using a solution that is running over Ethernet[1] but with custom hardware that is probably fairly expensive. They use a single time source and then measure the delay between each switch/node. Though, this is limited by needing to be able to run a cable between each node so idk how your definition of a distributed experiment fits. [1]: http://white-rabbit.web.cern.ch/ http://white-rabbit.web.cern.ch/
    - sethammons 4y ago
      We've several thousands of nodes in our system. Our ops people have witnessed clock skew from seconds to months that are transient. Time is constantly synced via ntpd. You can't trust system clocks in a distributed system to ensure ordering. Some reading: https://codeburst.io/why-shouldnt-you-trust-system-clocks-72a82a41df93 https://codeburst.io/why-shouldnt-you-trust-system-clocks-72...
  - dilyevsky 4y ago
    Drop the “over a network with unknown latency characteristics” - it’s cleaner
    - bjornsing 4y ago
      Cleaner perhaps, but wrong. If you know that latency is statistically symmetrical (same in both directions, on average) then you can synchronize clocks to arbitrary precision (asymptotically at least).
      - dilyevsky 3y ago
        > If you know that latency is statistically symmetrical (same in both directions, on average) My understanding of relativity is that this is principally unknowable and we only assume that speed of light is same in every direction by convention (every few years there’s a paper that claims they managed to measure one way sol but later it turns out they actually measured two-way in a roundabout way) so you can’t really know that?
- readams 4y ago
  You are definitely not within nanoseconds using NTP
  - withinboredom 4y ago
    I don’t think Chrony is NTP, if we are being pedantic, just NTP compatible. My servers have hardware timestamping on the network interface and chrony uses this.
    - rezonant 4y ago
      That would be PTP then
      - mlichvar 4y ago
        No, NTP and PTP are two different protocols. They can both use hardware timestamps and reach single-digit nanosecond accuracy in ideal conditions. The main difference is in existing support in switches and routers, which is needed to avoid the impact of asymmetric delay between ports (typically tens of nanoseconds per switch). PTP has good support in higher-end switches and routers, but it's difficult to secure and make resilient to failures. It was designed for automation and control networks in factories etc. NTP is a better fit for computer networks, but there doesn't seem to be any switches or routers with HW NTP support. If you really need the best accuracy with NTP, you can find old 100Mb/s hubs on ebay and create a separate network.
        rezonant 4y ago
        Yes, PTP and NTP are indeed very different protocols. There's no networking hardware timestamp support for NTP because NTP has nothing to do with hardware timestamps. PTP can be done without hardware timestamps, but it was designed with hardware support in mind. I don't know where you got it that NTP does anything even orders of magnitude close to nanoseconds: > NTP can usually maintain time to within tens of milliseconds over the public Internet, and can achieve better than one millisecond accuracy in local area networks under ideal conditions https://en.m.wikipedia.org/wiki/Network_Time_Protocol https://en.m.wikipedia.org/wiki/Network_Time_Protocol > The Precision Time Protocol (PTP) is a protocol used to synchronize clocks throughout a computer network. On a local area network, it achieves clock accuracy in the sub-microsecond range, making it suitable for measurement and control systems. https://en.m.wikipedia.org/wiki/Precision_Time_Protocol https://en.m.wikipedia.org/wiki/Precision_Time_Protocol Literally neither solution comes anywhere near nanosecond accuracy. For reference, there are 1,000,000 nanoseconds in a millisecond
- starfallg 4y ago
  In real world applications, NTP is accurate to the millisecond, while PTP is accurate to the microsecond. PTP is often used now in the Telco world as well as for broadcasting applications.
cbsmith 4y ago
I think the reality is making the SLA guarantees that Spanner in a third party cloud is just... not very practical.
- Cthulhu_ 4y ago
  That's it; there's plenty of competing products for Spanner, but Spanner's USP is the host and their hardware.
deepGem 4y ago
The atomic clock is not in the VM, it’s a service from what I understand similar to NTP but obviously much more localised. I in fact think once you offer atomic clock as a service for any public cloud, distributed transactions become a lot easier to implement thereby disrupting Spanner.
dangoodmanUT 4y ago
Last I checked the VMs are hooked up to their internal atomic clock time servers. They all have double digit microsecond drift at worst IIRC.

withinboredom 4y ago

Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other. It’s amazing what you can get, if you just ask. It’s the latter part you really want, which is doable with modern hardware accelerated time stamping and a peering NTP system (like chrony).

bjornsing 4y ago
Time synchronization over a network with unknown latency characteristics is fundamentally impossible.
- withinboredom 4y ago
  Wow. It’s a wonder anyone has a correct time then. /s Sarcasm aside, their NTP servers are less than 2 hops from my router. I know that because of excellent tools like ping and trace route.
  - rezonant 4y ago
    Based on your other post, it sounds like you are doing NTP with your ISP and then using PTP to sync your servers on the local network. Nothing about this implies that you are accurate to the global clock within a microsecond, let alone accurate within any number of nanoseconds. With a correctly equipped and configured PTP installation locally, you can expect your clocks to be synchronized on the order of a very small number of microseconds with respect to each other, but the relationship between those and the atomic global clock is something else. If you are using satellite/GPS time sources, then that's another matter. Why then is it relevant to have an NTP hookup at your ISP?
    - withinboredom 4y ago
      With A ISP, not my ISP. Also, I’m syncing with a Stratum 0 servers which are connected to an atomic clock and several GPS clocks. Also, consistency and accuracy are orthogonal to each other. My servers clocks are consistent and fairly accurate in regards to global time.
      - rezonant 4y ago
        Ah yes, I figured as much, this is why I skipped over the GPS and satellite parts initially. Having a high quality highly local NTP source is better than most sources, but it is nowhere near as close as having your own local time sources, which is what a lot of time sensitive installations do.
  - protocolture 4y ago
    no-propagate-ttl really does clean up an ISP network.
- ikiris 4y ago
  ptp to the ToR isn't rocket science. If you need something hyper precise and are a large enough customer, you can get just about anything you need built in even the big clouds. crazy customer needs is what made up most projects I saw or worked with.
- atoav 4y ago
  Really? It is now Tuesday in central europe and I guess most machines¹ managed to figure out that fact. Most machines will also know it is 8:11. The question is how much resolution do you need and are your clocks accuratly synced enough for the thing you plan to do. You are aware that many of the physics experiments that operate at the edge of what is possible use extremely accurate clocks synced over national and sometimes continental borders? ¹ let's ignore the ones that have been configured incorrectly
  - rezonant 4y ago
    Perfect time synchronization over a network with unknown latency is provably impossible. Fairly certain distributed physics experiments will be using atomic clocks that are not synchronized over the network. And/or they'll use direct satellite or GPS time sources which have predictable latencies.
    - t0b1 4y ago
      CERN at least seems to be using a solution that is running over Ethernet[1] but with custom hardware that is probably fairly expensive. They use a single time source and then measure the delay between each switch/node. Though, this is limited by needing to be able to run a cable between each node so idk how your definition of a distributed experiment fits. [1]: http://white-rabbit.web.cern.ch/ http://white-rabbit.web.cern.ch/
  - sethammons 4y ago
    We've several thousands of nodes in our system. Our ops people have witnessed clock skew from seconds to months that are transient. Time is constantly synced via ntpd. You can't trust system clocks in a distributed system to ensure ordering. Some reading: https://codeburst.io/why-shouldnt-you-trust-system-clocks-72a82a41df93 https://codeburst.io/why-shouldnt-you-trust-system-clocks-72...
- dilyevsky 4y ago
  Drop the “over a network with unknown latency characteristics” - it’s cleaner
  - bjornsing 4y ago
    Cleaner perhaps, but wrong. If you know that latency is statistically symmetrical (same in both directions, on average) then you can synchronize clocks to arbitrary precision (asymptotically at least).
    - dilyevsky 3y ago
      > If you know that latency is statistically symmetrical (same in both directions, on average) My understanding of relativity is that this is principally unknowable and we only assume that speed of light is same in every direction by convention (every few years there’s a paper that claims they managed to measure one way sol but later it turns out they actually measured two-way in a roundabout way) so you can’t really know that?
readams 4y ago
You are definitely not within nanoseconds using NTP
- withinboredom 4y ago
  I don’t think Chrony is NTP, if we are being pedantic, just NTP compatible. My servers have hardware timestamping on the network interface and chrony uses this.
  - rezonant 4y ago
    That would be PTP then
    - mlichvar 4y ago
      No, NTP and PTP are two different protocols. They can both use hardware timestamps and reach single-digit nanosecond accuracy in ideal conditions. The main difference is in existing support in switches and routers, which is needed to avoid the impact of asymmetric delay between ports (typically tens of nanoseconds per switch). PTP has good support in higher-end switches and routers, but it's difficult to secure and make resilient to failures. It was designed for automation and control networks in factories etc. NTP is a better fit for computer networks, but there doesn't seem to be any switches or routers with HW NTP support. If you really need the best accuracy with NTP, you can find old 100Mb/s hubs on ebay and create a separate network.
      - rezonant 4y ago
        Yes, PTP and NTP are indeed very different protocols. There's no networking hardware timestamp support for NTP because NTP has nothing to do with hardware timestamps. PTP can be done without hardware timestamps, but it was designed with hardware support in mind. I don't know where you got it that NTP does anything even orders of magnitude close to nanoseconds: > NTP can usually maintain time to within tens of milliseconds over the public Internet, and can achieve better than one millisecond accuracy in local area networks under ideal conditions https://en.m.wikipedia.org/wiki/Network_Time_Protocol https://en.m.wikipedia.org/wiki/Network_Time_Protocol > The Precision Time Protocol (PTP) is a protocol used to synchronize clocks throughout a computer network. On a local area network, it achieves clock accuracy in the sub-microsecond range, making it suitable for measurement and control systems. https://en.m.wikipedia.org/wiki/Precision_Time_Protocol https://en.m.wikipedia.org/wiki/Precision_Time_Protocol Literally neither solution comes anywhere near nanosecond accuracy. For reference, there are 1,000,000 nanoseconds in a millisecond
starfallg 4y ago
In real world applications, NTP is accurate to the millisecond, while PTP is accurate to the microsecond. PTP is often used now in the Telco world as well as for broadcasting applications.

cbsmith 4y ago

I think the reality is making the SLA guarantees that Spanner in a third party cloud is just... not very practical.

Cthulhu_ 4y ago
That's it; there's plenty of competing products for Spanner, but Spanner's USP is the host and their hardware.

deepGem 4y ago

The atomic clock is not in the VM, it’s a service from what I understand similar to NTP but obviously much more localised. I in fact think once you offer atomic clock as a service for any public cloud, distributed transactions become a lot easier to implement thereby disrupting Spanner.

dangoodmanUT 4y ago

Last I checked the VMs are hooked up to their internal atomic clock time servers. They all have double digit microsecond drift at worst IIRC.