6 ms·

> You design systems to make them self-correcting: Log events durably first, and then act on the events. If possible, make operations idempotent. If not, either

> You design systems to make them self-correcting: Log events durably first, and then act on the events. If possible, make operations idempotent. If not, either make operations fully commutative, or group them into larger commutative operations and use local transactions (so require local consistency) to ensure they are all applied at once. The point is not to require no atomicity, but to localise any guarantees as much as possible, because the more local they are, the less they affect throughput. You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. People often forget this: the user is sort of the unstated extra party in a distributed system. A canonical example is privacy settings on a social network. If I want to hide my profile from a user by unfriending them and then post something unkind about them, it doesn't really work for me that the application is free to reorder my actions. At some point your desire to have HA via a weakly consistent system conflicts with intuition and usability of an application.

vidarh 10y ago
> You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. The key being they don't expect their actions to be reordered. In most systems this is trivially handled by logging event time as seen from the users system and ensure the user sees a consistent view of their own operations. There are exceptions, naturally, but the point is not that you should be able to do this "in general", but that expecting global consistency is a relatively rare exception. If I post a comment here, I expect my comment to be visible when I return to the page. But there is no reason for me to notice if there is a strict global ordering to when posted comments first appears in the comment page. If someone presses submit on their post 100ms before me, but the comment first appears to me the next time I reload, while it appeared the opposite way for the other poster, it is extremely unlikely to matter. > A canonical example is privacy settings on a social network. If I want to hide my profile from a user by unfriending them and then post something unkind about them, it doesn't really for me that the application is free to reorder my actions. This is a design issue. Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements. As long as you do that, it makes no difference to you or anyone else if the setting and post was replicated globally in the same order or not.
- knucklesandwich 10y ago
  > Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements. I think you're misunderstanding what the problem is here. I want global consistency (really I want two party consistency) to prevent the said user from witnessing the unkind thing I posted about them. It's necessary that my own actions appear in their original order to me, but also to anyone else who is plausibly affected by their reordering. The full example also involves posting on a mutual friend's wall and having that post hidden by unfriending. You could route all requests to fetch posts to the owning poster's pinned set of servers, but this would be extremely expensive. Sharding solutions also weaken availability guarantees, because you necessarily shrink the set of nodes able to service a request.
  - eternalban 10y ago
    > I want two party consistency aka snapshot consistency. (Thoroughly enjoyed your "the user is sort of the unstated extra party in a distributed system", btw.)
  - wpietri 10y ago
    Do you really need global consistency for that? I think you can get by with user-consistent ordering as long as your data model's right. It sounds like you're saying that the friend-state for a given post are part of the permissions for the post, and therefore part of the post itself. In which case, when a user posts, I'd include the version number of their friend-state in the post's permissions. Then on view attempts, I make sure that the friend-state I'm working with is at least as new as the friend-state referred to by the post.
    - knucklesandwich 10y ago
      There are definitely solutions for handling this with consistency semantics that are something weaker than linearizable consistency. But they usually become expensive in other ways and loose a lot of the availability benefits to recover the consistency properties you do need. Most of the models we're even talking about here are something in the realm of sequential or causal consistency, which are still considered strong consistency models (and are usually implemented on top of linearizable algorithms that allow for weakening invariants that protect against witnessing inconsistency. spanner is an example of this). Your friend-state solution only protects the visibility of the post if the friend-state between the two users has not changed more recently than both the post and your version. I.E. Friend state 0 is we're friends. Friend state 1 is we're not friends. Friend state 2 is we're friends. Friend state 3 is we're not friends. I have friend state 2, and the post is at state 1. I accept my version incorrectly and am able to view the post, despite the fact that we're no longer friends.

vidarh 10y ago

> You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. The key being they don't expect their actions to be reordered. In most systems this is trivially handled by logging event time as seen from the users system and ensure the user sees a consistent view of their own operations. There are exceptions, naturally, but the point is not that you should be able to do this "in general", but that expecting global consistency is a relatively rare exception. If I post a comment here, I expect my comment to be visible when I return to the page. But there is no reason for me to notice if there is a strict global ordering to when posted comments first appears in the comment page. If someone presses submit on their post 100ms before me, but the comment first appears to me the next time I reload, while it appeared the opposite way for the other poster, it is extremely unlikely to matter. > A canonical example is privacy settings on a social network. If I want to hide my profile from a user by unfriending them and then post something unkind about them, it doesn't really for me that the application is free to reorder my actions. This is a design issue. Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements. As long as you do that, it makes no difference to you or anyone else if the setting and post was replicated globally in the same order or not.

knucklesandwich 10y ago
> Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements. I think you're misunderstanding what the problem is here. I want global consistency (really I want two party consistency) to prevent the said user from witnessing the unkind thing I posted about them. It's necessary that my own actions appear in their original order to me, but also to anyone else who is plausibly affected by their reordering. The full example also involves posting on a mutual friend's wall and having that post hidden by unfriending. You could route all requests to fetch posts to the owning poster's pinned set of servers, but this would be extremely expensive. Sharding solutions also weaken availability guarantees, because you necessarily shrink the set of nodes able to service a request.
- eternalban 10y ago
  > I want two party consistency aka snapshot consistency. (Thoroughly enjoyed your "the user is sort of the unstated extra party in a distributed system", btw.)
- wpietri 10y ago
  Do you really need global consistency for that? I think you can get by with user-consistent ordering as long as your data model's right. It sounds like you're saying that the friend-state for a given post are part of the permissions for the post, and therefore part of the post itself. In which case, when a user posts, I'd include the version number of their friend-state in the post's permissions. Then on view attempts, I make sure that the friend-state I'm working with is at least as new as the friend-state referred to by the post.
  - knucklesandwich 10y ago
    There are definitely solutions for handling this with consistency semantics that are something weaker than linearizable consistency. But they usually become expensive in other ways and loose a lot of the availability benefits to recover the consistency properties you do need. Most of the models we're even talking about here are something in the realm of sequential or causal consistency, which are still considered strong consistency models (and are usually implemented on top of linearizable algorithms that allow for weakening invariants that protect against witnessing inconsistency. spanner is an example of this). Your friend-state solution only protects the visibility of the post if the friend-state between the two users has not changed more recently than both the post and your version. I.E. Friend state 0 is we're friends. Friend state 1 is we're not friends. Friend state 2 is we're friends. Friend state 3 is we're not friends. I have friend state 2, and the post is at state 1. I accept my version incorrectly and am able to view the post, despite the fact that we're no longer friends.