5 ms·

As you say, the low-level operations in banking do provide atomicity, which are often lacking in distributed databases. Also, the banking system does have the

As you say, the low-level operations in banking do provide atomicity, which are often lacking in distributed databases. Also, the banking system does have the system of compensating transactions (e.g. returned checks) when constraints are violated, as well as an extensive system of auditing to maintain consistency. Do most distributed systems have the ability to detect inconsistency? How do they manage to correct problems when found? Is it mostly manual? I'm genuinely interested in hearing what people have done. I've certainly experienced inconsistencies which were in very low-stakes online situations that were still annoying. For example, a reddit comment that was posted on a thread, but never shows up my user page. Or when I stopped watching a movie on Netflix on my PC to go watch something else with my wife on our TV, but Netflix won't let us because it thinks I'm still watching on the PC. Come to think of it, there was also something that used to happen frequently with Amazon video, though I haven't seen it happen in a while. When I would resume a video, it gave me two choices of times to resume, one based on the device and one based on the server. The odd thing was that I had only ever used one device to watch the video but it was always the server-stored time that was right. Just a bug, I guess, but that does indicate the kind of inconsistencies that have to be resolved somehow.

vidarh 10y ago
> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that both or none steps gets durably recorded, and you may (or may not, depending on your requirements) insist on local ordering, but even there global consistency is unnecessary. > How do they manage to correct problems when found? Is it mostly manual? I'm genuinely interested in hearing what people have done. You design systems to make them self-correcting: Log events durably first, and then act on the events. If possible, make operations idempotent. If not, either make operations fully commutative, or group them into larger commutative operations and use local transactions (so require local consistency) to ensure they are all applied at once. The point is not to require no atomicity, but to localise any guarantees as much as possible, because the more local they are, the less they affect throughput. On failure, your system needs to be able to determine which operations have been applied, and which have not, and resume processing of operations that have not. > I've certainly experienced inconsistencies which were in very low-stakes online situations that were still annoying. For example, a reddit comment that was posted on a thread, but never shows up my user page. Or when I stopped watching a movie on Netflix on my PC to go watch something else with my wife on our TV, but Netflix won't let us because it thinks I'm still watching on the PC. This type of thing is often down to copping out on guarantees when a system includes multiple data stores, and is often fixed by applying a local transactional event log used to trigger jobs that are guaranteed to get processed. E.g. consider the Reddit example - if you want to be able to effortlessly shard, you might denormalize and store data about the comment both tied to a thread, and tid to a user. If these are in two different databases, you need some form of distributed transactional system. A simple way of achieving that is to write a transactional log locally on each system (as long as operations are design to not need coordination) that ensures that both "attach post to thread" and "attach post to user page" will get logged successfully before giving a success response to the end user. You can do that by using a commit marker in your log. Then you "just" need a job processor that marks jobs done when all operations in a transaction has completed (and that can handle re-running the same ones). But this is something people often fail to do, because it's easier to just write out the code for each step sequentially and deal with the rare instances where a server dies in the middle of processing a request. So the basic pattern for ditching global consistency is to write transaction logs locally, representing each set of operations you need to apply, and put in place a mechanism to ensure each one is applied no matter what. Then use that "everywhere" including in the many places where you should've had guarantees like that, but don't because developers have been lazy.
- knucklesandwich 10y ago
  > You design systems to make them self-correcting: Log events durably first, and then act on the events. If possible, make operations idempotent. If not, either make operations fully commutative, or group them into larger commutative operations and use local transactions (so require local consistency) to ensure they are all applied at once. The point is not to require no atomicity, but to localise any guarantees as much as possible, because the more local they are, the less they affect throughput. You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. People often forget this: the user is sort of the unstated extra party in a distributed system. A canonical example is privacy settings on a social network. If I want to hide my profile from a user by unfriending them and then post something unkind about them, it doesn't really work for me that the application is free to reorder my actions. At some point your desire to have HA via a weakly consistent system conflicts with intuition and usability of an application.
  - vidarh 10y ago
    > You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. The key being they don't expect their actions to be reordered. In most systems this is trivially handled by logging event time as seen from the users system and ensure the user sees a consistent view of their own operations. There are exceptions, naturally, but the point is not that you should be able to do this "in general", but that expecting global consistency is a relatively rare exception. If I post a comment here, I expect my comment to be visible when I return to the page. But there is no reason for me to notice if there is a strict global ordering to when posted comments first appears in the comment page. If someone presses submit on their post 100ms before me, but the comment first appears to me the next time I reload, while it appeared the opposite way for the other poster, it is extremely unlikely to matter. > A canonical example is privacy settings on a social network. If I want to hide my profile from a user by unfriending them and then post something unkind about them, it doesn't really for me that the application is free to reorder my actions. This is a design issue. Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements. As long as you do that, it makes no difference to you or anyone else if the setting and post was replicated globally in the same order or not.

vidarh 10y ago

> Do most distributed systems have the ability to detect inconsistency? They do if you build them to have them. Your bank account works without requiring consistency because every operation is commutative: You never do anything that says "set account to value x". You have operations like "deposit x" and "withdraw x". For operations that require you move money from one account to another, you need to guarantee that both or none steps gets durably recorded, and you may (or may not, depending on your requirements) insist on local ordering, but even there global consistency is unnecessary. > How do they manage to correct problems when found? Is it mostly manual? I'm genuinely interested in hearing what people have done. You design systems to make them self-correcting: Log events durably first, and then act on the events. If possible, make operations idempotent. If not, either make operations fully commutative, or group them into larger commutative operations and use local transactions (so require local consistency) to ensure they are all applied at once. The point is not to require no atomicity, but to localise any guarantees as much as possible, because the more local they are, the less they affect throughput. On failure, your system needs to be able to determine which operations have been applied, and which have not, and resume processing of operations that have not. > I've certainly experienced inconsistencies which were in very low-stakes online situations that were still annoying. For example, a reddit comment that was posted on a thread, but never shows up my user page. Or when I stopped watching a movie on Netflix on my PC to go watch something else with my wife on our TV, but Netflix won't let us because it thinks I'm still watching on the PC. This type of thing is often down to copping out on guarantees when a system includes multiple data stores, and is often fixed by applying a local transactional event log used to trigger jobs that are guaranteed to get processed. E.g. consider the Reddit example - if you want to be able to effortlessly shard, you might denormalize and store data about the comment both tied to a thread, and tid to a user. If these are in two different databases, you need some form of distributed transactional system. A simple way of achieving that is to write a transactional log locally on each system (as long as operations are design to not need coordination) that ensures that both "attach post to thread" and "attach post to user page" will get logged successfully before giving a success response to the end user. You can do that by using a commit marker in your log. Then you "just" need a job processor that marks jobs done when all operations in a transaction has completed (and that can handle re-running the same ones). But this is something people often fail to do, because it's easier to just write out the code for each step sequentially and deal with the rare instances where a server dies in the middle of processing a request. So the basic pattern for ditching global consistency is to write transaction logs locally, representing each set of operations you need to apply, and put in place a mechanism to ensure each one is applied no matter what. Then use that "everywhere" including in the many places where you should've had guarantees like that, but don't because developers have been lazy.

knucklesandwich 10y ago
> You design systems to make them self-correcting: Log events durably first, and then act on the events. If possible, make operations idempotent. If not, either make operations fully commutative, or group them into larger commutative operations and use local transactions (so require local consistency) to ensure they are all applied at once. The point is not to require no atomicity, but to localise any guarantees as much as possible, because the more local they are, the less they affect throughput. You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. People often forget this: the user is sort of the unstated extra party in a distributed system. A canonical example is privacy settings on a social network. If I want to hide my profile from a user by unfriending them and then post something unkind about them, it doesn't really work for me that the application is free to reorder my actions. At some point your desire to have HA via a weakly consistent system conflicts with intuition and usability of an application.
- vidarh 10y ago
  > You can't do this in general, because we often want to build applications where the user expects their actions won't be reordered. The key being they don't expect their actions to be reordered. In most systems this is trivially handled by logging event time as seen from the users system and ensure the user sees a consistent view of their own operations. There are exceptions, naturally, but the point is not that you should be able to do this "in general", but that expecting global consistency is a relatively rare exception. If I post a comment here, I expect my comment to be visible when I return to the page. But there is no reason for me to notice if there is a strict global ordering to when posted comments first appears in the comment page. If someone presses submit on their post 100ms before me, but the comment first appears to me the next time I reload, while it appeared the opposite way for the other poster, it is extremely unlikely to matter. > A canonical example is privacy settings on a social network. If I want to hide my profile from a user by unfriending them and then post something unkind about them, it doesn't really for me that the application is free to reorder my actions. This is a design issue. Your privacy settings is a local concern to your account, it most certainly does not need to be globally consistent. All you need to do to avoid problems with that is to ensure consistency in routing requests for a given session, and enforce local consistency of ordering within a given user account or session depending on requirements. As long as you do that, it makes no difference to you or anyone else if the setting and post was replicated globally in the same order or not.