4 ms·

You don't actually need absolute time for distributed systems to work, only independently verifiable order. Independent venerability can be achieved through Mer

You don't actually need absolute time for distributed systems to work, only independently verifiable order. Independent venerability can be achieved through Merkle structures and Merkle proofs. In a way, through the proofs you can communicate anybody you're "perspective" on the order of events. If you get several "perspectives" you can therefore independently infer the absolute order of events. You would still be left with race conditions between the communicating nodes, but that is something you can't get around anyway.

eternalban 11y ago
The deductive approach to system ordering (e.g. via computing hashes) sets a fairly high floor on latency. If you have deep pockets like Google, you will opt for the expensive GPS/Atomic Clock and work within the epsilon bounds.
- justicezyx 11y ago
  The atomic clock is intended to bound timing skew, not latency exactly. As long as clocks at different places are synced with high precision, you could still sync operations within the epsilon bounds, despite a huge delay. Of course, that requires the system to be design accordingly to accommodate the huge latency.
  - hinkley 11y ago
    I was always impressed that the HTTP 1.0 spec had this problem mostly solved back in the early 90's, and several times I've had to remind colleagues of this fact. They knew that the machines would never agree what time "now" is so all interactions contain time stamps from both parties. If you want to know what time the server has right now, you can get it from the Response headers on your last communication (i.e., often the one you're decoding right now). If the server says this reply expires at noon but you think it's 1:15 already, you can sort out that the server really means "five minutes from now" and is 1:20 behind of you for some reason. I'm starting to think at a properly constructed time library should contain local and remote time instead of trying to find an objective truth. It's too easy to transpose. [edit: even I invert my clock skews from time to time.]
  - eternalban 11y ago
    Late reply but noted this now. We disagree. Perfect clock obviates the need for protocol based consensus mechanism. Paxos & friends have substantial latency costs. Secondary effects of protocol based approach include NAK storms, reduction of available bandwidth (consumed by the chatty consensus protocol packets). Tertiary effects include triggering of congestion control mechanisms. There is zero question that a distributed system built on high fidelity clocks will mop the proverbial floor in terms of performance.
    - jasonwatkinspdx 11y ago
      A couple responses: Even with the hardware, google can't shrink the window past 7ms or so, based on published reports. To preserve consistency there are situations where you need to wait out the clock uncertainty. Spanner still uses paxos for replication because ordering is only part of the problem consensus solves.
deleted 11y ago
[deleted]
Retric 11y ago
At that point you might as well just have larger ticks and let events occur at the same time. If you really want strict ordering you can always impose arbitrary ordering. Ex: all events in tick X occur in the order each machine decides, and then order by machine IP address.
hinkley 11y ago
I've been wondering about this lately, and I'm pretty sure on several projects we've used the order of event to imply causal relationships. Now I'm thinking we should have modeled cause and effect and only used the dates when communicating with the customer (I bought what, when??). But I'm not entirely sure what that looks like.

eternalban 11y ago

The deductive approach to system ordering (e.g. via computing hashes) sets a fairly high floor on latency. If you have deep pockets like Google, you will opt for the expensive GPS/Atomic Clock and work within the epsilon bounds.

justicezyx 11y ago
The atomic clock is intended to bound timing skew, not latency exactly. As long as clocks at different places are synced with high precision, you could still sync operations within the epsilon bounds, despite a huge delay. Of course, that requires the system to be design accordingly to accommodate the huge latency.
- hinkley 11y ago
  I was always impressed that the HTTP 1.0 spec had this problem mostly solved back in the early 90's, and several times I've had to remind colleagues of this fact. They knew that the machines would never agree what time "now" is so all interactions contain time stamps from both parties. If you want to know what time the server has right now, you can get it from the Response headers on your last communication (i.e., often the one you're decoding right now). If the server says this reply expires at noon but you think it's 1:15 already, you can sort out that the server really means "five minutes from now" and is 1:20 behind of you for some reason. I'm starting to think at a properly constructed time library should contain local and remote time instead of trying to find an objective truth. It's too easy to transpose. [edit: even I invert my clock skews from time to time.]
- eternalban 11y ago
  Late reply but noted this now. We disagree. Perfect clock obviates the need for protocol based consensus mechanism. Paxos & friends have substantial latency costs. Secondary effects of protocol based approach include NAK storms, reduction of available bandwidth (consumed by the chatty consensus protocol packets). Tertiary effects include triggering of congestion control mechanisms. There is zero question that a distributed system built on high fidelity clocks will mop the proverbial floor in terms of performance.
  - jasonwatkinspdx 11y ago
    A couple responses: Even with the hardware, google can't shrink the window past 7ms or so, based on published reports. To preserve consistency there are situations where you need to wait out the clock uncertainty. Spanner still uses paxos for replication because ordering is only part of the problem consensus solves.

deleted 11y ago

[deleted]

Retric 11y ago

At that point you might as well just have larger ticks and let events occur at the same time. If you really want strict ordering you can always impose arbitrary ordering. Ex: all events in tick X occur in the order each machine decides, and then order by machine IP address.

hinkley 11y ago

I've been wondering about this lately, and I'm pretty sure on several projects we've used the order of event to imply causal relationships. Now I'm thinking we should have modeled cause and effect and only used the dates when communicating with the customer (I bought what, when??). But I'm not entirely sure what that looks like.