4 ms·

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system.

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production workloads which no MQ system can support.

dxxvi 1y ago
Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?
- deleted 1y ago
  [deleted]
- saberience 1y ago
  Think about this for a second. Kafka offsets are a thing, consumer groups are a thing. It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ since you can always re-read messages once they have been processed, whereas generally with MQ systems the message is then marked for deletion and asynchronously deleted.
  - throwwgisgreat 1y ago
    > better behaviour than RabbitMQ since you can always re-read messages once they have been processed I can imagine, a 1 Billion dollar transaction accidentally gets processed by ten thousand client nodes due to a client app synchronization bug, company rethinks its dumb data dumper server strategy...news at 11.
  - otterley 1y ago
    > It's trivial to ensure that only one message is delivered to only one consumer Exactly-once delivery is one of the hardest distributed systems problems. If you’ve “trivially” solved it, please show us your solution.
    - bccdee 1y ago
      > It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ The trivial solution is to use Kafka. They're clearly saying that Kafka makes it trivial, not that it's trivial to solve from scratch.
      - deleted 1y ago
        [deleted]
      - otterley 1y ago
        What the parent poster described isn’t what makes Kafka’s “exactly once” semantics work. It’s the use of an idempotency token associated with each publication, which effectively turns “at-least-once” semantics into effectively “exactly once” via deduplication.
        saberience 1y ago
        We used Flink and S3 compatible backends (for the Flink state) to ensure exactly once processing. I didn't say Kafka magically solves these problems for you, but it was required for the scalability we needed.
- LgWoodenBadger 1y ago
  It "disappears" in the sense that the Consumer-Group that read/committed that message (event) will never see it again. It doesn't "disappear" in the sense that a new Consumer-Group can be started in a way that will get that message, or you can reset your Consumer-Group's offset to re-consume it.
guhcampos 1y ago
It is a *very bad* replacement for an MQ system, for the simple reason you can't quickly and effortlessly scale int/out consumers.
- saberience 1y ago
  Why can't you? In my experience, scaling Kafka was far easier than scaling our RabbitMQ cluster. We started running into issues when our RabbitMQ cluster hit 25k TPS, our Kafka cluster of equivalent resources didn't break a sweat at 500k TPS.
  - otterley 1y ago
    Scaling consumers, not throughput. And it’s both directions (in and out, not just out).
  - jdswain 1y ago
    What sort of systems do you work on to require this kind of traffic volume? I've worked on one project that I'd consider relatively high volume (UK Post Office Horizon Online) and we were only targeting 500 TPS.
    - saberience 1y ago
      Think extremely popular multiplayer videogames. We used these systems for all the backend infra, logins, logouts, chat messages, purchases. We often had millions of players online at a given moment which means lots of transactions!
  - guhcampos 1y ago
    How did you handle re-partitioning and rebalancing every time you scaled your cluster in or out?
rileymichael 1y ago
> you can make it work just as well as any MQ based system you really can't. getting per-message acks, dynamically scaling competing consumers without having to repartition while retaining ordering, etc. requires a ton of hacks like client side tracking / building your own storage on top of offset metadata / etc.. and you still won't have all of the features actual message queues provide. to make it worse, there is very little public work/discussion on this so you'll be on your own to figure out all of the quirks. the only notable example is https://github.com/confluentinc/parallel-consumer https://github.com/confluentinc/parallel-consumer which is effectively abandoned
- LgWoodenBadger 1y ago
  I don't think "while retaining ordering" is something you want to include here, since you can't guarantee processing order for any MQ system without serializing the consumption down to a single consumer.
  - rileymichael 1y ago
    absolute order, you're correct. but several systems support demultiplexing an interleaved stream by some key so that order is retained for all messages in each key space (pulsar key_shared subscriptions, azure service bus sessions, etc). you're still serializing consumption / have an exclusive consumer for each key, but with global parallelism. here's the equivalent in parallel-consumer: https://github.com/confluentinc/parallel-consumer?tab=readme-ov-file#ordered-by-key https://github.com/confluentinc/parallel-consumer?tab=readme...
- mwarkentin 1y ago
  There is a whole KIP that is “preview” on Kafka 4.1 to handle this use case natively: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=255070434#content/view/255070434 https://cwiki.apache.org/confluence/plugins/servlet/mobile?c... Note: I haven’t had a chance to test it out in anger personally yet.
  - rileymichael 1y ago
gmokki 1y ago
Don't you always need a database after reading events from Kafka to deduplication? So the competing solutions are: PostgreSQL or Kafka+PostgreSQL Kafka does provide some extrs there, handling load spikes, more clients that PG can handle natively and resilience to some DB downtime. But is it worth the complexity, in most cases no.
- jevyjevjevs 1y ago
  Actually you can avoid having a separate DB! You can build a materialized view of the data using [KTables](https://developer.confluent.io/courses/kafka-streams/ktable/ https://developer.confluent.io/courses/kafka-streams/ktable/) or use [interactive queries](https://developer.confluent.io/courses/kafka-streams/interactive-queries/ https://developer.confluent.io/courses/kafka-streams/interac...). The "table" is built up from a backing kafka topic so you don't need maintain another datastore if the data view you want is entirely derived from one or more Kafka topics.
  - enether 1y ago
    My understanding is this is pretty niche and can be complex.

dxxvi 1y ago

Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?

deleted 1y ago
[deleted]
saberience 1y ago
Think about this for a second. Kafka offsets are a thing, consumer groups are a thing. It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ since you can always re-read messages once they have been processed, whereas generally with MQ systems the message is then marked for deletion and asynchronously deleted.
- throwwgisgreat 1y ago
  > better behaviour than RabbitMQ since you can always re-read messages once they have been processed I can imagine, a 1 Billion dollar transaction accidentally gets processed by ten thousand client nodes due to a client app synchronization bug, company rethinks its dumb data dumper server strategy...news at 11.
- otterley 1y ago
  > It's trivial to ensure that only one message is delivered to only one consumer Exactly-once delivery is one of the hardest distributed systems problems. If you’ve “trivially” solved it, please show us your solution.
  - bccdee 1y ago
    > It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ The trivial solution is to use Kafka. They're clearly saying that Kafka makes it trivial, not that it's trivial to solve from scratch.
    - deleted 1y ago
      [deleted]
    - otterley 1y ago
      What the parent poster described isn’t what makes Kafka’s “exactly once” semantics work. It’s the use of an idempotency token associated with each publication, which effectively turns “at-least-once” semantics into effectively “exactly once” via deduplication.
      - saberience 1y ago
        We used Flink and S3 compatible backends (for the Flink state) to ensure exactly once processing. I didn't say Kafka magically solves these problems for you, but it was required for the scalability we needed.
LgWoodenBadger 1y ago
It "disappears" in the sense that the Consumer-Group that read/committed that message (event) will never see it again. It doesn't "disappear" in the sense that a new Consumer-Group can be started in a way that will get that message, or you can reset your Consumer-Group's offset to re-consume it.

guhcampos 1y ago

It is a *very bad* replacement for an MQ system, for the simple reason you can't quickly and effortlessly scale int/out consumers.

saberience 1y ago
Why can't you? In my experience, scaling Kafka was far easier than scaling our RabbitMQ cluster. We started running into issues when our RabbitMQ cluster hit 25k TPS, our Kafka cluster of equivalent resources didn't break a sweat at 500k TPS.
- otterley 1y ago
  Scaling consumers, not throughput. And it’s both directions (in and out, not just out).
- jdswain 1y ago
  What sort of systems do you work on to require this kind of traffic volume? I've worked on one project that I'd consider relatively high volume (UK Post Office Horizon Online) and we were only targeting 500 TPS.
  - saberience 1y ago
    Think extremely popular multiplayer videogames. We used these systems for all the backend infra, logins, logouts, chat messages, purchases. We often had millions of players online at a given moment which means lots of transactions!
- guhcampos 1y ago
  How did you handle re-partitioning and rebalancing every time you scaled your cluster in or out?

rileymichael 1y ago

> you can make it work just as well as any MQ based system you really can't. getting per-message acks, dynamically scaling competing consumers without having to repartition while retaining ordering, etc. requires a ton of hacks like client side tracking / building your own storage on top of offset metadata / etc.. and you still won't have all of the features actual message queues provide. to make it worse, there is very little public work/discussion on this so you'll be on your own to figure out all of the quirks. the only notable example is https://github.com/confluentinc/parallel-consumer https://github.com/confluentinc/parallel-consumer which is effectively abandoned

LgWoodenBadger 1y ago
I don't think "while retaining ordering" is something you want to include here, since you can't guarantee processing order for any MQ system without serializing the consumption down to a single consumer.
- rileymichael 1y ago
  absolute order, you're correct. but several systems support demultiplexing an interleaved stream by some key so that order is retained for all messages in each key space (pulsar key_shared subscriptions, azure service bus sessions, etc). you're still serializing consumption / have an exclusive consumer for each key, but with global parallelism. here's the equivalent in parallel-consumer: https://github.com/confluentinc/parallel-consumer?tab=readme-ov-file#ordered-by-key https://github.com/confluentinc/parallel-consumer?tab=readme...
mwarkentin 1y ago
There is a whole KIP that is “preview” on Kafka 4.1 to handle this use case natively: https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=255070434#content/view/255070434 https://cwiki.apache.org/confluence/plugins/servlet/mobile?c... Note: I haven’t had a chance to test it out in anger personally yet.
- rileymichael 1y ago

gmokki 1y ago

Don't you always need a database after reading events from Kafka to deduplication? So the competing solutions are: PostgreSQL or Kafka+PostgreSQL Kafka does provide some extrs there, handling load spikes, more clients that PG can handle natively and resilience to some DB downtime. But is it worth the complexity, in most cases no.

jevyjevjevs 1y ago
Actually you can avoid having a separate DB! You can build a materialized view of the data using [KTables](https://developer.confluent.io/courses/kafka-streams/ktable/ https://developer.confluent.io/courses/kafka-streams/ktable/) or use [interactive queries](https://developer.confluent.io/courses/kafka-streams/interactive-queries/ https://developer.confluent.io/courses/kafka-streams/interac...). The "table" is built up from a backing kafka topic so you don't need maintain another datastore if the data view you want is entirely derived from one or more Kafka topics.
- enether 1y ago
  My understanding is this is pretty niche and can be complex.