4 ms·

> any reasonable datacenter operation is going to detect clock skew beyond some tolerance (order of seconds) It takes time to recognize such failures. If the c

> any reasonable datacenter operation is going to detect clock skew beyond some tolerance (order of seconds) It takes time to recognize such failures. If the clock skew is only off by a few seconds or even minutes, the impact isn't so horrible, but if the clock jumps ahead by 20 years, you've effectively prevented any writes to that key for 20 years. Even if you only consider a few days, that's long enough that it can result in lost business. > logically nondeterministic state timestamps should only be used to tiebreak between logically observed events. You've somehow combined a timestamp field as a tiebreaker, as client data to be stored, and as a request id. If you separate them then adjusting the timestamp as a tiebreaker (along with node and counter) will remain logically consistent. > And it's important to zoom out and look at the actual consequences of missed writes at the application layer Lets. You wrote a distributed data store that can lose writes under certain circumstances. If the UX impact is negligible for you, that's great. However, it may be catastrophic for another use (say storing the latest price read for a stock option). For developers to use your database, it means they need to do an impact analysis for every single use. Just wait until someone decides the API is easy enough to use that they try to put payment data in there.

deleted 7y ago
[deleted]
peterbourgon 7y ago
> If the clock skew is only off by a few seconds or even minutes, the impact isn't so horrible, but if the clock jumps ahead by 20 years, you've effectively prevented any writes to that key for 20 years. That's true, but (luckily) it's easy to remediate those objects in the Roshi architecture. > If the UX impact [of lost writes] is negligible for you, that's great. One interesting thing that I've observed about this kind of data system work is that there seems to be a as-of-yet unsolvable tension between "general purpose applicability" and "specific purpose usability". That is, there isn't a way to design a CRDT-based causally-consistent data system which has a general purpose API that consumers can use for a suitably broad set of use cases. Instead, success stories like Roshi seem to require specializing both data semantics and APIs to the specific use cases they target. So, yes, if someone can't tolerate/remediate lost writes due to clock drift, then Roshi is not a suitable system. But, that's fine: data systems don't have to be general purpose to be useful, or to advance the state of the art.
- marknadal 7y ago
  General purpose CRDTs have been available for at least 5+ years. We have such a system ( http://github.com/amark/gun http://github.com/amark/gun ) that is running in production at the Internet Archive, and P2P versions of Reddit & YouTube that have ~2M users. Dealing with this UX stuff is very important, and we have distributed systems correctness tests (https://github.com/gundb/panic-server https://github.com/gundb/panic-server) that does split brain simulation and recovery, and code to handle timestamp skew even if NTP is running. If you pass this off to developers to be responsible for, they won't know the difference between mistakes in your system and mistakes in theirs. They shouldn't need to be a distributed systems expert to use the database.
- specialist 7y ago
  I'm really digging this thread. Nicely done. Thanks. As a noob, some kind of Starting Assumptions and Context would be cool for all these distributed infra projects. For instance, I wouldn't have known about mitigating time skews in data centers. Or perhaps even a reference deployment, image(s), playbook, or whatever the devops people are now calling it.

deleted 7y ago

[deleted]

peterbourgon 7y ago

> If the clock skew is only off by a few seconds or even minutes, the impact isn't so horrible, but if the clock jumps ahead by 20 years, you've effectively prevented any writes to that key for 20 years. That's true, but (luckily) it's easy to remediate those objects in the Roshi architecture. > If the UX impact [of lost writes] is negligible for you, that's great. One interesting thing that I've observed about this kind of data system work is that there seems to be a as-of-yet unsolvable tension between "general purpose applicability" and "specific purpose usability". That is, there isn't a way to design a CRDT-based causally-consistent data system which has a general purpose API that consumers can use for a suitably broad set of use cases. Instead, success stories like Roshi seem to require specializing both data semantics and APIs to the specific use cases they target. So, yes, if someone can't tolerate/remediate lost writes due to clock drift, then Roshi is not a suitable system. But, that's fine: data systems don't have to be general purpose to be useful, or to advance the state of the art.

marknadal 7y ago
General purpose CRDTs have been available for at least 5+ years. We have such a system ( http://github.com/amark/gun http://github.com/amark/gun ) that is running in production at the Internet Archive, and P2P versions of Reddit & YouTube that have ~2M users. Dealing with this UX stuff is very important, and we have distributed systems correctness tests (https://github.com/gundb/panic-server https://github.com/gundb/panic-server) that does split brain simulation and recovery, and code to handle timestamp skew even if NTP is running. If you pass this off to developers to be responsible for, they won't know the difference between mistakes in your system and mistakes in theirs. They shouldn't need to be a distributed systems expert to use the database.
specialist 7y ago
I'm really digging this thread. Nicely done. Thanks. As a noob, some kind of Starting Assumptions and Context would be cool for all these distributed infra projects. For instance, I wouldn't have known about mitigating time skews in data centers. Or perhaps even a reference deployment, image(s), playbook, or whatever the devops people are now calling it.