4 ms·

It's not databases that are hard to scale, it's consistent state. It's just that databases begin with the design assumption that you want state to be consisten

It's not databases that are hard to scale, it's consistent state. It's just that databases begin with the design assumption that you want state to be consistent.

naasking 10y ago
This is a great point. I think the way forward is to unite these properties in an append-only schema that can be efficiently queried for a consistent state without requiring expensive transactions. Something like [1] works well enough for RDMSs, but still isn't ideal. [1] http://higherlogics.blogspot.ca/2015/10/versioning-domain-entities.html http://higherlogics.blogspot.ca/2015/10/versioning-domain-en...
- zaphar 10y ago
  This is also not as easy as it sounds though when you go distributed. Databases already handle the non distributed case pretty well. It's when you need the performance/latency benefits of distribution that you begin to run into problems that an append only schema doesn't help with.
- takeda 10y ago
  IMO one of great things that came out of NoSQL are CRDTs[1]. This is something that would be interesting to see if it could work with a relational database. [1] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...
  - naasking 10y ago
    Concurrent Revisions [1] are even better IMO. Much simpler to reason about. In fact, if you know how to work with git and other distributed SCM tools, then you already understand the model: branch, do your work locally, then deterministic merge. [1] http://research.microsoft.com/en-us/projects/revisions/ http://research.microsoft.com/en-us/projects/revisions/
    - Jweb_Guru 10y ago
      Yes, I just found out about these recently and am surprised no datastores have been built around them. They are basically a usable version of fork-join causal consistency. They also subsume CRDTs.
      - naasking 10y ago
        Do they? The papers on CRDTs I've skimmed often mention concurrent revisions, but they've never claimed CR subsume CRDTs.
        Jweb_Guru 10y ago
        This paper does (in fact, it points out that any total function can be emulated by a query-update automaton, which can be used with concurrent revisions). At least, that's my recollection, it could be a generalization of CR.
        naasking 10y ago
        What is the "this paper" you're referring to? The papers at the link I provided don't mention CRDTs much.
        Jweb_Guru 10y ago
        http://research.microsoft.com/pubs/155638/msr-tr-2011-117%282%29.pdf http://research.microsoft.com/pubs/155638/msr-tr-2011-117%28...: "Our definition is strictly more general than CRDTs [17] in the following sense: From any state-based CRDT we can obtain a FJ-QUA by using the same state and initial state, the same query and update functions, a fork function that creates a new replica and then merges the forker state, and a join function that uses the merge. Note that the definition of strong eventual consistency in [17], just like ours, requires that updates can be applied to any state."
        jacques_chester 10y ago

It's not databases that are hard to scale, it's consistent state. It's just that databases begin with the design assumption that you want state to be consistent.

naasking 10y ago

This is a great point. I think the way forward is to unite these properties in an append-only schema that can be efficiently queried for a consistent state without requiring expensive transactions. Something like [1] works well enough for RDMSs, but still isn't ideal. [1] http://higherlogics.blogspot.ca/2015/10/versioning-domain-entities.html http://higherlogics.blogspot.ca/2015/10/versioning-domain-en...

zaphar 10y ago
This is also not as easy as it sounds though when you go distributed. Databases already handle the non distributed case pretty well. It's when you need the performance/latency benefits of distribution that you begin to run into problems that an append only schema doesn't help with.
takeda 10y ago
IMO one of great things that came out of NoSQL are CRDTs[1]. This is something that would be interesting to see if it could work with a relational database. [1] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...
- naasking 10y ago
  Concurrent Revisions [1] are even better IMO. Much simpler to reason about. In fact, if you know how to work with git and other distributed SCM tools, then you already understand the model: branch, do your work locally, then deterministic merge. [1] http://research.microsoft.com/en-us/projects/revisions/ http://research.microsoft.com/en-us/projects/revisions/
  - Jweb_Guru 10y ago
    Yes, I just found out about these recently and am surprised no datastores have been built around them. They are basically a usable version of fork-join causal consistency. They also subsume CRDTs.
    - naasking 10y ago
      Do they? The papers on CRDTs I've skimmed often mention concurrent revisions, but they've never claimed CR subsume CRDTs.
      - Jweb_Guru 10y ago
        This paper does (in fact, it points out that any total function can be emulated by a query-update automaton, which can be used with concurrent revisions). At least, that's my recollection, it could be a generalization of CR.
        naasking 10y ago
        What is the "this paper" you're referring to? The papers at the link I provided don't mention CRDTs much.
        Jweb_Guru 10y ago
        http://research.microsoft.com/pubs/155638/msr-tr-2011-117%282%29.pdf http://research.microsoft.com/pubs/155638/msr-tr-2011-117%28...: "Our definition is strictly more general than CRDTs [17] in the following sense: From any state-based CRDT we can obtain a FJ-QUA by using the same state and initial state, the same query and update functions, a fork function that creates a new replica and then merges the forker state, and a join function that uses the merge. Note that the definition of strong eventual consistency in [17], just like ours, requires that updates can be applied to any state."
        jacques_chester 10y ago