3 ms·

Introducing Incremental Consistency Guarantees

29 points by adizere 10y ago

eternalban 10y ago
My read of this is that the authors believe they have a "novel" candidate set of universal semantics and an API to encapsulate what seems to be well known patterns in this space. I think this approach is viable in conjunction with reusable components and radically swappable backends: ICG would be the non-moving interface that (per the claims) would allow perfect reuse without modification due to changed storage model. However the typical consumer of products in this space is building high performance and tightly integrated systems. And in terms of contributions to formal definitions, it seems they are introducing neologisms that fairly match one to one to existing terminology and concepts. > To program with ICG, applications need to wait asynchronously for multiple replies to an operation (where each reply encapsulates a different guarantee on the result) while doing useful work, i.e., speculate. To the best of our knowledge, no abstraction fulfills these criteria. [1] Operating on (possibly stale) snapshots of an MVCC store? Making changes to a local copy (e.g. pull) of a blockchain (e.g. git)? > Correctables Merge conflict? > Speculate Optimistic commit? [1]: referenced preprint at https://arxiv.org/pdf/1609.02434v1.pdf https://arxiv.org/pdf/1609.02434v1.pdf
- adizere 10y ago
  Indeed, high performance and integrated systems form a big chunk of the intended "consumer space" of Correctables. But being integrated or high perf. does not mean these systems do not use language-level constructs; what I have in mind, as example of constructs, are mainly Promises, which is the root abstraction under Correctables. Regarding MVCC: speculation there is done, by default, by the concurrency control system, so applications remain unaware that their state is, at any point, speculative. In other words, the application does not do any useful work while waiting for something, but it's the concurrency control system which cleverly overlaps (and sometimes undoes) transactions. Also, the applications always observes a single consistency model (isolation) at play. Agree with you that working on a git branch is the same as doing useful work on a preliminary view in the ICG model. Git (and POW-based blockchains likewise) are quite some peculiar applications though, aren't they? How many applications -- I'm thinking here mainly about web services -- do we see thriving in this disconnected model of work, where forks are inherently prone to appear? I can imagine many wouldn't be happy to see their e-mail inbox constantly forking and showing them various versions, occasionally dropping e-mails, etc. Or you shopping cart acting like it had multiple personalities. :) I would also add, as food for thought, that there is a technical bit of difference between how ICG is meant to operate and how request-replies usually operate, which might be relevant for some applications: in ICG, ideally, there should be a single request sent to the service, and that request would resolve in multiple steps, until eventually reaching the correct (strongly-consistent) view. Not going to insist on this point though, since it's a more complicated matter which is not too critical. Also, I'm not sure what you mean exactly by candidate set of universal semantics; I'd be glad to continue our discussion on this path, however. adrian // one of the co-authors
  - eternalban 10y ago
    [Sorry for the late replay, Adrian, in case you read this.] > Candidate set of universal semantics more precisely the semantics and api for a uniform treatment of the full state[-transition] space of an Eventually Consistent Available Partition Tolerant (ECAP) distributed system. (I would think you will agree that ICG is fully bounded by ECAP and does not concern CP systems. The dual to ICG would be an incremental availability guarantees. I'll grant that these two modalities (ICG/IAG) seem to have a wave/particle type of relationship.) The concern with performance hit is mainly that, would the conceptual framework prevent categories of optimization.
    - adizere 10y ago
      Hey eternalban, thanks, very interesting comments! It's plain to see that you have a deeper / more theoretical perspective on distsys than I tend to have. Would you describe universal semantics as being "Turing-complete in the context of a distributed system"? One way to look at it is that Correctables are essentially just wrappers over traditional RPCs. So in this sense they would enable universal semantics -- you can treat the whole space of transitions in a distributed system. Now it's not clear to me why you need the make the distinction between CP and AP. ICG is orthogonal to these categories. Consider the following example. Say you're using MySQL (a CP system) and you also have a client-side cache (essentially AP). A Correctable can supply results from either or both of these systems: if it's either, then it will manifest as an AP or as an CP system (depending where the result arrives from -- you can choose); if it's both, then you'll essentially get an AP system, since it's possible that your result from MySQL is forsaken due to a partition. Does this make sense? > re: IAG /\ ICG Incremental availability guarantees I'd say it's a bit of a misnomer, since availability is not such a shaded guarantee as consistency is. I regard availability as a black & white property, whereas consistency is a whole spectrum of colours -- even more, which can be represented as a directed acyclic graph. Even so, the concept of IAG would have some practical consequences. I'll now run the risk of taking it maybe a bit too far: but I think whenever an application chooses a consistency model, that model inherently comes along with its own preset availability guarantee. As consistency models get stronger, there is an instant switch from available to non-available models. [I should cite some refs here but I'm in a hurry.] Consequently, if you have incremental consistency guarantees then you also have decreasing availability guarantees. Also, your analogy to wave-particle duality is very interesting, though it's a shame that it's above my head to ponder over it. I'll read more about it to see if I can connect the dots.

29 points by adizere 10y ago

My read of this is that the authors believe they have a "novel" candidate set of universal semantics and an API to encapsulate what seems to be well known patterns in this space. I think this approach is viable in conjunction with reusable components and radically swappable backends: ICG would be the non-moving interface that (per the claims) would allow perfect reuse without modification due to changed storage model. However the typical consumer of products in this space is building high performance and tightly integrated systems. And in terms of contributions to formal definitions, it seems they are introducing neologisms that fairly match one to one to existing terminology and concepts. > To program with ICG, applications need to wait asynchronously for multiple replies to an operation (where each reply encapsulates a different guarantee on the result) while doing useful work, i.e., speculate. To the best of our knowledge, no abstraction fulfills these criteria. [1] Operating on (possibly stale) snapshots of an MVCC store? Making changes to a local copy (e.g. pull) of a blockchain (e.g. git)? > Correctables Merge conflict? > Speculate Optimistic commit? [1]: referenced preprint at https://arxiv.org/pdf/1609.02434v1.pdf https://arxiv.org/pdf/1609.02434v1.pdf

adizere 10y ago
Indeed, high performance and integrated systems form a big chunk of the intended "consumer space" of Correctables. But being integrated or high perf. does not mean these systems do not use language-level constructs; what I have in mind, as example of constructs, are mainly Promises, which is the root abstraction under Correctables. Regarding MVCC: speculation there is done, by default, by the concurrency control system, so applications remain unaware that their state is, at any point, speculative. In other words, the application does not do any useful work while waiting for something, but it's the concurrency control system which cleverly overlaps (and sometimes undoes) transactions. Also, the applications always observes a single consistency model (isolation) at play. Agree with you that working on a git branch is the same as doing useful work on a preliminary view in the ICG model. Git (and POW-based blockchains likewise) are quite some peculiar applications though, aren't they? How many applications -- I'm thinking here mainly about web services -- do we see thriving in this disconnected model of work, where forks are inherently prone to appear? I can imagine many wouldn't be happy to see their e-mail inbox constantly forking and showing them various versions, occasionally dropping e-mails, etc. Or you shopping cart acting like it had multiple personalities. :) I would also add, as food for thought, that there is a technical bit of difference between how ICG is meant to operate and how request-replies usually operate, which might be relevant for some applications: in ICG, ideally, there should be a single request sent to the service, and that request would resolve in multiple steps, until eventually reaching the correct (strongly-consistent) view. Not going to insist on this point though, since it's a more complicated matter which is not too critical. Also, I'm not sure what you mean exactly by candidate set of universal semantics; I'd be glad to continue our discussion on this path, however. adrian // one of the co-authors
- eternalban 10y ago
  [Sorry for the late replay, Adrian, in case you read this.] > Candidate set of universal semantics more precisely the semantics and api for a uniform treatment of the full state[-transition] space of an Eventually Consistent Available Partition Tolerant (ECAP) distributed system. (I would think you will agree that ICG is fully bounded by ECAP and does not concern CP systems. The dual to ICG would be an incremental availability guarantees. I'll grant that these two modalities (ICG/IAG) seem to have a wave/particle type of relationship.) The concern with performance hit is mainly that, would the conceptual framework prevent categories of optimization.
  - adizere 10y ago
    Hey eternalban, thanks, very interesting comments! It's plain to see that you have a deeper / more theoretical perspective on distsys than I tend to have. Would you describe universal semantics as being "Turing-complete in the context of a distributed system"? One way to look at it is that Correctables are essentially just wrappers over traditional RPCs. So in this sense they would enable universal semantics -- you can treat the whole space of transitions in a distributed system. Now it's not clear to me why you need the make the distinction between CP and AP. ICG is orthogonal to these categories. Consider the following example. Say you're using MySQL (a CP system) and you also have a client-side cache (essentially AP). A Correctable can supply results from either or both of these systems: if it's either, then it will manifest as an AP or as an CP system (depending where the result arrives from -- you can choose); if it's both, then you'll essentially get an AP system, since it's possible that your result from MySQL is forsaken due to a partition. Does this make sense? > re: IAG /\ ICG Incremental availability guarantees I'd say it's a bit of a misnomer, since availability is not such a shaded guarantee as consistency is. I regard availability as a black & white property, whereas consistency is a whole spectrum of colours -- even more, which can be represented as a directed acyclic graph. Even so, the concept of IAG would have some practical consequences. I'll now run the risk of taking it maybe a bit too far: but I think whenever an application chooses a consistency model, that model inherently comes along with its own preset availability guarantee. As consistency models get stronger, there is an instant switch from available to non-available models. [I should cite some refs here but I'm in a hurry.] Consequently, if you have incremental consistency guarantees then you also have decreasing availability guarantees. Also, your analogy to wave-particle duality is very interesting, though it's a shame that it's above my head to ponder over it. I'll read more about it to see if I can connect the dots.