4 ms·

> you can make it work just as well as any MQ based system you really can't. getting per-message acks, dynamically scaling competing consumers without having t

> you can make it work just as well as any MQ based system you really can't. getting per-message acks, dynamically scaling competing consumers without having to repartition while retaining ordering, etc. requires a ton of hacks like client side tracking / building your own storage on top of offset metadata / etc.. and you still won't have all of the features actual message queues provide. to make it worse, there is very little public work/discussion on this so you'll be on your own to figure out all of the quirks. the only notable example is https://github.com/confluentinc/parallel-consumer https://github.com/confluentinc/parallel-consumer which is effectively abandoned

LgWoodenBadger 11mo ago
I don't think "while retaining ordering" is something you want to include here, since you can't guarantee processing order for any MQ system without serializing the consumption down to a single consumer.
- rileymichael 11mo ago
  absolute order, you're correct. but several systems support demultiplexing an interleaved stream by some key so that order is retained for all messages in each key space (pulsar key_shared subscriptions, azure service bus sessions, etc). you're still serializing consumption / have an exclusive consumer for each key, but with global parallelism. here's the equivalent in parallel-consumer: https://github.com/confluentinc/parallel-consumer?tab=readme-ov-file#ordered-by-key https://github.com/confluentinc/parallel-consumer?tab=readme...
mwarkentin 11mo ago
There is a whole KIP that is “preview” on Kafka 4.1 to handle this use case natively: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=255070434#content/view/255070434 https://cwiki.apache.org/confluence/plugins/servlet/mobile?c... Note: I haven’t had a chance to test it out in anger personally yet.
- rileymichael 11mo ago
  yep i mentioned that in another thread below. very excited to see it added, but it will be some time until its production ready / they implement the full feature set

LgWoodenBadger 11mo ago

I don't think "while retaining ordering" is something you want to include here, since you can't guarantee processing order for any MQ system without serializing the consumption down to a single consumer.

rileymichael 11mo ago
absolute order, you're correct. but several systems support demultiplexing an interleaved stream by some key so that order is retained for all messages in each key space (pulsar key_shared subscriptions, azure service bus sessions, etc). you're still serializing consumption / have an exclusive consumer for each key, but with global parallelism. here's the equivalent in parallel-consumer: https://github.com/confluentinc/parallel-consumer?tab=readme-ov-file#ordered-by-key https://github.com/confluentinc/parallel-consumer?tab=readme...

mwarkentin 11mo ago

There is a whole KIP that is “preview” on Kafka 4.1 to handle this use case natively: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=255070434#content/view/255070434 https://cwiki.apache.org/confluence/plugins/servlet/mobile?c... Note: I haven’t had a chance to test it out in anger personally yet.

rileymichael 11mo ago
yep i mentioned that in another thread below. very excited to see it added, but it will be some time until its production ready / they implement the full feature set