5 ms·

You need to think about what makes messages distinct. Content addressing will help you as long as you have a rigorous definition of identity, which you might ot

by btrask 11y ago

You need to think about what makes messages distinct. Content addressing will help you as long as you have a rigorous definition of identity, which you might otherwise be doing ad-hoc when you assign UUIDs. If the same message is meaningful at different points in time, your hashed content should include a timestamp (and of course you need to choose an appropriate level of precision). If your message is meaningful in different contents, it needs to reference that context somehow (ideally by hash). The payoff for doing this well is 1. an elimination of double-sent emails, double-posts, etc. (when the software might not know if it went through the first time), and 2. the ability to deduplicate across a network partition, for example if a user makes the same change on their computer and on their phone and then syncs them.

zAy0LfpBZLC8mAC 11y ago
... except wallclock timestamps for ordering of events in a distributed system is a terrible idea in most cases for the simple reason that (a) it's not a (even close to) guaranteed unique event identifier (the same time happens in many locations), (b) you usually don't have a clock of sufficient precision to even reliably establish a total order of distinct moments and (c) even if you have, that order does not necessarily preserve causality. Now, UUIDs don't solve (b) or (c) either, but at least they actually give you a reliable identifier for events.
- btrask 11y ago
  No one is talking about ordering events.
  - schrodinger 11y ago
    Pretty sure messages are ordered events in this scenario, no?
    - btrask 11y ago
      They're events, but no ordering is assumed, guaranteed, promised, implied, or enforced. The timestamps, if you choose to use them, are only for determining message identity (equality) and thus can be as accurate or inaccurate as relevant for your application. You don't have to use timestamps if they don't make sense. For example, what is the "identity" of a message in an instant messaging program? If you just use the message content, then duplicate messages don't work. It would make sense to include the sender and the time written (which is different from the time sent!) as well. You might also include the recipient, depending. Note that instead of using timestamps, you could use the hash of the message being replied to. However, IM software usually doesn't have a system for real replies, so doing that properly probably isn't worthwhile. (Edit: however, hashes are very useful for systems like email, forums, etc., where reply targets are explicitly defined.) If you do want ordering, BYO. Content addressing and UUIDs are equivalent in that regard.
      - zAy0LfpBZLC8mAC 11y ago
        > The timestamps, if you choose to use them, are only for determining message identity (equality) Which simply doesn't work because timestamps are not unique to location. Every clock experiences every timestamp, and thus potentially assigns any timestamp to an event, so multiple clocks can end up assigning the same timestamp to distinct events, and thus identifying them. > Note that instead of using timestamps, you could use the hash of the message being replied to. If your message isn't unique, the message you are replying to probably isn't either, so it's useless to use it as an identifier. > If you do want ordering, BYO. Content addressing and UUIDs are equivalent in that regard. No, they are not, at all, they aren't even the same kind of thing. UUIDs are unique identifiers, you generate them to label things that couldn't be distinguished without them. Content addressing is nothing more than summarizing a given object in a way that (almost certainly) preserves identity. So, one creates distinction, the other maintains identity.
        btrask 11y ago
        > Which simply doesn't work because timestamps are not unique to location. Yes, I know. This is okay. Whatever time you think you wrote the message at is fine. Even if your clock is off. Even if you lie. That is (part of) the message content (in this case). The time the message is sent is meaningless and can't be used, I agree. The time the message was written, or created, or "thought to be created" is what matters (or doesn't). > If your message isn't unique, the message you are replying to probably isn't either, so it's useless to use it as an identifier. Git does this and works great. If your entire message history (via hash chain) is identical, then you really do have duplicate messages. > UUIDs are unique identifiers, you generate them to label things that couldn't be distinguished without them. Content addressing is nothing more than summarizing a given object in a way that (almost certainly) preserves identity. So, one creates distinction, the other maintains identity. I agree, you're right. However, this is a feature.
  - zAy0LfpBZLC8mAC 11y ago
    ... in which case my post explains why UUIDs are the better solution.
    - btrask 11y ago
      Content hashing doesn't rely on timestamps to act as a unique identifier. It relies on all of the content, which may (or may not) include a timestamp, depending on your application. To the degree that collisions occur, that's a feature, not a bug. If you get undesirable collisions, you aren't hashing the right data. Edit: to clarify, content hashes are just like UUIDs except you can get useful collisions if you want them.
- nhaehnle 11y ago
  The context here is events originated by a single actor, so wallclock is much more reasonable than in the general distributed system case. You might still be worried about the user accessing your system simultaneously from multiple devices, so a randomly generated ID is probably still useful.
  - zAy0LfpBZLC8mAC 11y ago
    First, it's usually impossible to tell actors apart, so it's not particularly useful to build an algorithm that depends on doing so. Second, while the actor might be the same, the time source usually is not, so it still doesn't help you. Third, if actor and time source are indeed the same, you really are using a counter, the fact that it's correlated with wallclock time is coincidental.
    - philsnow 11y ago
      Piling on, even if the actor and time source are the same, you have to be really careful about how your use "wall clock time", because if you're just asking the OS what time it is, a lot of platforms allow the clock to move backwards, not move, etc.

by btrask 11y ago

zAy0LfpBZLC8mAC 11y ago

... except wallclock timestamps for ordering of events in a distributed system is a terrible idea in most cases for the simple reason that (a) it's not a (even close to) guaranteed unique event identifier (the same time happens in many locations), (b) you usually don't have a clock of sufficient precision to even reliably establish a total order of distinct moments and (c) even if you have, that order does not necessarily preserve causality. Now, UUIDs don't solve (b) or (c) either, but at least they actually give you a reliable identifier for events.

btrask 11y ago
No one is talking about ordering events.
- schrodinger 11y ago
  Pretty sure messages are ordered events in this scenario, no?
  - btrask 11y ago
    They're events, but no ordering is assumed, guaranteed, promised, implied, or enforced. The timestamps, if you choose to use them, are only for determining message identity (equality) and thus can be as accurate or inaccurate as relevant for your application. You don't have to use timestamps if they don't make sense. For example, what is the "identity" of a message in an instant messaging program? If you just use the message content, then duplicate messages don't work. It would make sense to include the sender and the time written (which is different from the time sent!) as well. You might also include the recipient, depending. Note that instead of using timestamps, you could use the hash of the message being replied to. However, IM software usually doesn't have a system for real replies, so doing that properly probably isn't worthwhile. (Edit: however, hashes are very useful for systems like email, forums, etc., where reply targets are explicitly defined.) If you do want ordering, BYO. Content addressing and UUIDs are equivalent in that regard.
    - zAy0LfpBZLC8mAC 11y ago
      > The timestamps, if you choose to use them, are only for determining message identity (equality) Which simply doesn't work because timestamps are not unique to location. Every clock experiences every timestamp, and thus potentially assigns any timestamp to an event, so multiple clocks can end up assigning the same timestamp to distinct events, and thus identifying them. > Note that instead of using timestamps, you could use the hash of the message being replied to. If your message isn't unique, the message you are replying to probably isn't either, so it's useless to use it as an identifier. > If you do want ordering, BYO. Content addressing and UUIDs are equivalent in that regard. No, they are not, at all, they aren't even the same kind of thing. UUIDs are unique identifiers, you generate them to label things that couldn't be distinguished without them. Content addressing is nothing more than summarizing a given object in a way that (almost certainly) preserves identity. So, one creates distinction, the other maintains identity.
      - btrask 11y ago
        > Which simply doesn't work because timestamps are not unique to location. Yes, I know. This is okay. Whatever time you think you wrote the message at is fine. Even if your clock is off. Even if you lie. That is (part of) the message content (in this case). The time the message is sent is meaningless and can't be used, I agree. The time the message was written, or created, or "thought to be created" is what matters (or doesn't). > If your message isn't unique, the message you are replying to probably isn't either, so it's useless to use it as an identifier. Git does this and works great. If your entire message history (via hash chain) is identical, then you really do have duplicate messages. > UUIDs are unique identifiers, you generate them to label things that couldn't be distinguished without them. Content addressing is nothing more than summarizing a given object in a way that (almost certainly) preserves identity. So, one creates distinction, the other maintains identity. I agree, you're right. However, this is a feature.
- zAy0LfpBZLC8mAC 11y ago
  ... in which case my post explains why UUIDs are the better solution.
  - btrask 11y ago
    Content hashing doesn't rely on timestamps to act as a unique identifier. It relies on all of the content, which may (or may not) include a timestamp, depending on your application. To the degree that collisions occur, that's a feature, not a bug. If you get undesirable collisions, you aren't hashing the right data. Edit: to clarify, content hashes are just like UUIDs except you can get useful collisions if you want them.
nhaehnle 11y ago
The context here is events originated by a single actor, so wallclock is much more reasonable than in the general distributed system case. You might still be worried about the user accessing your system simultaneously from multiple devices, so a randomly generated ID is probably still useful.
- zAy0LfpBZLC8mAC 11y ago
  First, it's usually impossible to tell actors apart, so it's not particularly useful to build an algorithm that depends on doing so. Second, while the actor might be the same, the time source usually is not, so it still doesn't help you. Third, if actor and time source are indeed the same, you really are using a counter, the fact that it's correlated with wallclock time is coincidental.
  - philsnow 11y ago
    Piling on, even if the actor and time source are the same, you have to be really careful about how your use "wall clock time", because if you're just asking the OS what time it is, a lot of platforms allow the clock to move backwards, not move, etc.