8 ms·

Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap

Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...

manigandham 6y ago
Why? It's just exposing the partitions underneath (which all horizontally scaled event/log systems use) and letting you order by the key within each partition. This is exactly like Kafk, Pulsar, AWS Kinesis, Azure Eventhubs, and many others which are all being used by thousands of applications without issue.
- kamalaboulhosn 6y ago
  Actually, it's not exactly exposing the partition underneath. The ordering key is the only unit a user has to deal with: think user ID or database row primary key. Throughput is limited to 1MB/s per ordering key, but the number of ordering keys is limited only by what can be represented in a 1KB string. One can have millions of keys if desired and they don't need to be known in advance. The system scales automatically as needed, so there is no need to think in terms of partitions or worry about repartitioning when load increases. Order is preserved through this scaling without the user having to do anything. Disclaimer: I work on Cloud Pub/Sub and am the primary engineer on the ordering feature.
  - manigandham 6y ago
    It's hashing the ordering key to map to a partition right? That's what I meant, access to ordering via partitions but without the full addressing of named partitions like Kafka.
    - kamalaboulhosn 6y ago
      There is a mapping from ordering key to internal storage structures, yes. The goal is that users are abstracted away from any notion of partition and the system handles scaling and ordering on its own, e.g., when there are hot shards.
stingraycharles 6y ago
Elaborate please, why is receiving messages in an ordered fashion a foot gun and a trap? Isn’t it totally acceptable to rely on this type of behavior if your use case demands it?
- royjacobs 6y ago
  Because this will break in situations where you have multiple consumers or if there's any sort of redelivery/failure scenario. Ordering only works if there's a single-threaded consumer somewhere that processes exactly one message at a time and blocks processing as soon as there is an error of any kind. The amount of scenarios that this works for is probably very small. The footgun is that people don't realize the above constraints and assume that ordering will magically fix a lot of ordering issues (some more examples of this in the rest of this comments section).
- vp8989 6y ago
  One way it can be a foot gun is: To ensure ordering, systems like these have to block consumption on any given partition until it gets an ACK. If publishers of the data don't understand that, they can publish with too coarse a partition key which greatly decreases the amount of concurrent consumption you can have. Not knocking how these systems work, it's more that people generally don't "read the fine print", especially these days with cloud services.
jlokier 6y ago
Technically I agree, but I think in most cases, the alternative is the endpoints need to handle unordered messages correctly and that won't happen either. So which is better: The queue does it, hiding subtle footguns because doing distributed systems reliably is hard; or the queue doesn't do it and the applications hanging off have failures more often (but not often enough for people to fix) because distributed systems reliability is still hard.
- jeffbee 6y ago
  All of the uses of Kafka I have encountered in my professional career have been deployed simply to make something asynchronous, that is, kafka exists to be a low-impedance sink of things to do later in some other process. I've not run across one that actually needed those items to be processed in order. Therefore it is my impression that ordering can be a false requirement.
  - vp8989 6y ago
    Agreed, also "processing in order" != "consuming from the sink in order". If everything upstream and downstream of the sink doesn't also happen in order then using an ordered sink is pointless.

Seriously, I can't shake the feeling that most people who want this are likely very, very wrong and that this "feature" will turn out to be a footgun and a trap more often than a sniper rifle...

manigandham 6y ago

Why? It's just exposing the partitions underneath (which all horizontally scaled event/log systems use) and letting you order by the key within each partition. This is exactly like Kafk, Pulsar, AWS Kinesis, Azure Eventhubs, and many others which are all being used by thousands of applications without issue.

kamalaboulhosn 6y ago
Actually, it's not exactly exposing the partition underneath. The ordering key is the only unit a user has to deal with: think user ID or database row primary key. Throughput is limited to 1MB/s per ordering key, but the number of ordering keys is limited only by what can be represented in a 1KB string. One can have millions of keys if desired and they don't need to be known in advance. The system scales automatically as needed, so there is no need to think in terms of partitions or worry about repartitioning when load increases. Order is preserved through this scaling without the user having to do anything. Disclaimer: I work on Cloud Pub/Sub and am the primary engineer on the ordering feature.
- manigandham 6y ago
  It's hashing the ordering key to map to a partition right? That's what I meant, access to ordering via partitions but without the full addressing of named partitions like Kafka.
  - kamalaboulhosn 6y ago
    There is a mapping from ordering key to internal storage structures, yes. The goal is that users are abstracted away from any notion of partition and the system handles scaling and ordering on its own, e.g., when there are hot shards.

stingraycharles 6y ago

Elaborate please, why is receiving messages in an ordered fashion a foot gun and a trap? Isn’t it totally acceptable to rely on this type of behavior if your use case demands it?

royjacobs 6y ago
Because this will break in situations where you have multiple consumers or if there's any sort of redelivery/failure scenario. Ordering only works if there's a single-threaded consumer somewhere that processes exactly one message at a time and blocks processing as soon as there is an error of any kind. The amount of scenarios that this works for is probably very small. The footgun is that people don't realize the above constraints and assume that ordering will magically fix a lot of ordering issues (some more examples of this in the rest of this comments section).
vp8989 6y ago
One way it can be a foot gun is: To ensure ordering, systems like these have to block consumption on any given partition until it gets an ACK. If publishers of the data don't understand that, they can publish with too coarse a partition key which greatly decreases the amount of concurrent consumption you can have. Not knocking how these systems work, it's more that people generally don't "read the fine print", especially these days with cloud services.

jlokier 6y ago

Technically I agree, but I think in most cases, the alternative is the endpoints need to handle unordered messages correctly and that won't happen either. So which is better: The queue does it, hiding subtle footguns because doing distributed systems reliably is hard; or the queue doesn't do it and the applications hanging off have failures more often (but not often enough for people to fix) because distributed systems reliability is still hard.

jeffbee 6y ago
All of the uses of Kafka I have encountered in my professional career have been deployed simply to make something asynchronous, that is, kafka exists to be a low-impedance sink of things to do later in some other process. I've not run across one that actually needed those items to be processed in order. Therefore it is my impression that ordering can be a false requirement.
- vp8989 6y ago
  Agreed, also "processing in order" != "consuming from the sink in order". If everything upstream and downstream of the sink doesn't also happen in order then using an ordered sink is pointless.