5 ms·

I've never really understood the appeal of ordered messages. You end up splitting your data across partitions anyways for parallelism, so who cares? What system

I've never really understood the appeal of ordered messages. You end up splitting your data across partitions anyways for parallelism, so who cares? What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle.

geodel 6y ago
Exactly. We ran into same issue with Kafka. If one needs ordered messages for some reason, Kafka is pretty much useless for this.
mrkeen 6y ago
> You end up splitting your data across partitions anyways Messages are ordered within partitions. > What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle. TCP/IP ?
- staticassertion 6y ago
  > Messages are ordered within partitions. Right, but that means you're still "unordered" across those partitions? > TCP/IP ? But TCP/IP isn't delivered in order, it rearranges the unordered packages by their ID. I guess ordered delivery would be nice for that, but I just feel like making your protocol not require ordering is far simpler. Not to mention that both TCP and Kafka have to handle head of line blocking? I'm not trying to say that ordering is bad or anything, I just feel like it isn't buying me tons.
  - lmm 6y ago
    > Right, but that means you're still "unordered" across those partitions? Right, so related messages have an ordering guarantee but unrelated messages may be processed out of order relative to each other, which is usually what you want. (Of course you do have to set the record key correctly). > I'm not trying to say that ordering is bad or anything, I just feel like it isn't buying me tons. It's a lot more lightweight than full ACID, but if you get your dataflow right it achieves everything that a traditional database does. Without ordering you wouldn't be able to do anything that requires any kind of consistency.
    - staticassertion 6y ago
      Hm, ok, yeah. So I guess I can see what you mean. I've never had a use case where I felt comfortable relying on any kind of message ordering, and always rely on my application level logic to handle that, or ensure the system is resilient despite ordering (ie: commutative operations only). To me, it seemed at odds with the parallelism of a partition, but I suppose in this case you'd be partitioning on some sort of semantic key vs, say, a hash. Thanks for bearing with me on that, this was just an unfamiliar idea for me.
  - mrkeen 6y ago
    > But TCP/IP isn't delivered in order.. I guess ordered delivery would be nice > both TCP and Kafka have to handle head of line blocking Well which is it? (If TCP doesn't give you ordered delivery, why would a head block the rest of the line?)
C4stor 6y ago
You can split with a business case in mind. Maybe if you're an e-business, you'll split everything happening on your website by client id, but still want events belonging to a single client to be received in order, for practicality.

geodel 6y ago

Exactly. We ran into same issue with Kafka. If one needs ordered messages for some reason, Kafka is pretty much useless for this.

mrkeen 6y ago

> You end up splitting your data across partitions anyways Messages are ordered within partitions. > What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle. TCP/IP ?

staticassertion 6y ago
> Messages are ordered within partitions. Right, but that means you're still "unordered" across those partitions? > TCP/IP ? But TCP/IP isn't delivered in order, it rearranges the unordered packages by their ID. I guess ordered delivery would be nice for that, but I just feel like making your protocol not require ordering is far simpler. Not to mention that both TCP and Kafka have to handle head of line blocking? I'm not trying to say that ordering is bad or anything, I just feel like it isn't buying me tons.
- lmm 6y ago
  > Right, but that means you're still "unordered" across those partitions? Right, so related messages have an ordering guarantee but unrelated messages may be processed out of order relative to each other, which is usually what you want. (Of course you do have to set the record key correctly). > I'm not trying to say that ordering is bad or anything, I just feel like it isn't buying me tons. It's a lot more lightweight than full ACID, but if you get your dataflow right it achieves everything that a traditional database does. Without ordering you wouldn't be able to do anything that requires any kind of consistency.
  - staticassertion 6y ago
    Hm, ok, yeah. So I guess I can see what you mean. I've never had a use case where I felt comfortable relying on any kind of message ordering, and always rely on my application level logic to handle that, or ensure the system is resilient despite ordering (ie: commutative operations only). To me, it seemed at odds with the parallelism of a partition, but I suppose in this case you'd be partitioning on some sort of semantic key vs, say, a hash. Thanks for bearing with me on that, this was just an unfamiliar idea for me.
- mrkeen 6y ago
  > But TCP/IP isn't delivered in order.. I guess ordered delivery would be nice > both TCP and Kafka have to handle head of line blocking Well which is it? (If TCP doesn't give you ordered delivery, why would a head block the rest of the line?)

C4stor 6y ago

You can split with a business case in mind. Maybe if you're an e-business, you'll split everything happening on your website by client id, but still want events belonging to a single client to be received in order, for practicality.