8 ms·

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement f

Semantic but important point, Kafka is not a queue, it's a distributed append only log. I deal with so many people who think it's a super-scalable replacement for an MQ, and it's such the wrong way to think about it.

codeflo 1y ago
Do you mean this in the sense that listeners don't remove messages, as one would expect from a queue data structure?
- EdwardDiego 1y ago
  Exactly. There's no concept in Kafka (yet...) of "acking" or DLQs, Kafka is very good at what it does by being deliberately stupid, it knows nothing about your messages or who has consumed them and who hasn't. That was all deliberately pushed onto consumers to manage to achieve scale.
  - speed_spread 1y ago
    Kafka is the MongoDB of sequential storage. Built for webscale and then widely adopted based on impressive single-metric numbers without regard for actual fitness to purpose in smaller operations. Fortunately it always what reliable enough. I believe RabbitMQ is much more balanced and is closer to what people expect from a high level queueing/pubsub system.
  - LgWoodenBadger 1y ago
    What do you mean "no concept...of asking?" During consumption, you must either auto-commit offsets, or manually commit them. If you don't, you'll get the same events over and over again.
    - deleted 1y ago
      [deleted]
    - EdwardDiego 1y ago
      Or you can just store your next offset in a DB and tell the consumer to read from that offset - the Kafka offset storage topic is a convenient implementation of the most common usage pattern, but it's not mandatory to use - and again, the broker doesn't do anything with that information when serving you data. Acking in an MQ is very different.
- jen20 1y ago
  That is the major difference - clients track their read offsets rather than the structure removing messages. There aren't really "listeners" in the sense of a pub-sub?
  - munchbunny 1y ago
    > There aren't really "listeners" in the sense of a pub-sub? They are still "listeners", it's that events aren't pushed to the listener. Instead the API is designed for sheer throughput.
- nasretdinov 1y ago
  Well, it's impractical to try to handle messages individually in Kafka, it's designed to acknowledge entire batches (since it's a distributed append-only log). You can still do that, but the performance will be no better than an SQL database
BerislavLopac 1y ago
To be fair, any (immutable) data structure that includes the creation timestamp can be a queue. It might not be a good queue, but it can be used as one.
- hamandcheese 1y ago
  On this note... has anyone here used object storage directly as a queue? How did it go? You can make a bucket immutable, and entries have timestamps. I don't think any cloud provider makes claims about the accuracy or monoatomicity of these timestamps, so you would merely get an ordering, not necessarily the ordering in which things truly occurred. But I have use cases where that is fine. I believe with a clever naming scheme and cooperating clients it could be made to work.
  - dvhh 1y ago
    Once used object storage as queue, you can implement queue semantic at the application level, with one object per entry. But the application was fairly low volume in Data and usage, So eventual consistency and capacity was not an issue. And yes timestamp monotonicity is not guaranteed when multiple client upload at the same time so unique id was given to each client at startup and used for to add guarantee of entries name. Metadata and prefix were used to indicate state of object during processing. Not ideal, but it was cheaper that a DB or a dedicated MQ. The application did not last, but would try again the approach if adapted to stuation.
    - hamandcheese 1y ago
      The application I'm interested in is a log-based artifact registry. Volume would be very low. Much more important is the immutability and durability of the log. I was thinking that writes could be indexed/prefixed into timestamp buckets according to the clients local time. This can't be trusted, of course. But the application consumers could detect and reject any writes whose upload timestamp exceeds a fixed delta from the timestamp bucket it was uploaded to. That allows for arbitrary seeking to any point on the log.
saberience 1y ago
Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production workloads which no MQ system can support.
- dxxvi 1y ago
  Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?
  - deleted 1y ago
    [deleted]
  - saberience 1y ago
    Think about this for a second. Kafka offsets are a thing, consumer groups are a thing. It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ since you can always re-read messages once they have been processed, whereas generally with MQ systems the message is then marked for deletion and asynchronously deleted.
    - throwwgisgreat 1y ago
      > better behaviour than RabbitMQ since you can always re-read messages once they have been processed I can imagine, a 1 Billion dollar transaction accidentally gets processed by ten thousand client nodes due to a client app synchronization bug, company rethinks its dumb data dumper server strategy...news at 11.
    - otterley 1y ago
      > It's trivial to ensure that only one message is delivered to only one consumer Exactly-once delivery is one of the hardest distributed systems problems. If you’ve “trivially” solved it, please show us your solution.
munchbunny 1y ago
Yup, that's a fair and important distinction, Kafka comes with very little of the ergonomics that make MQ good at what it does. In exchange you get blistering scale, for when blistering scale is the main problem.

codeflo 1y ago

Do you mean this in the sense that listeners don't remove messages, as one would expect from a queue data structure?

EdwardDiego 1y ago
Exactly. There's no concept in Kafka (yet...) of "acking" or DLQs, Kafka is very good at what it does by being deliberately stupid, it knows nothing about your messages or who has consumed them and who hasn't. That was all deliberately pushed onto consumers to manage to achieve scale.
- speed_spread 1y ago
  Kafka is the MongoDB of sequential storage. Built for webscale and then widely adopted based on impressive single-metric numbers without regard for actual fitness to purpose in smaller operations. Fortunately it always what reliable enough. I believe RabbitMQ is much more balanced and is closer to what people expect from a high level queueing/pubsub system.
- LgWoodenBadger 1y ago
  What do you mean "no concept...of asking?" During consumption, you must either auto-commit offsets, or manually commit them. If you don't, you'll get the same events over and over again.
  - deleted 1y ago
    [deleted]
  - EdwardDiego 1y ago
    Or you can just store your next offset in a DB and tell the consumer to read from that offset - the Kafka offset storage topic is a convenient implementation of the most common usage pattern, but it's not mandatory to use - and again, the broker doesn't do anything with that information when serving you data. Acking in an MQ is very different.
jen20 1y ago
That is the major difference - clients track their read offsets rather than the structure removing messages. There aren't really "listeners" in the sense of a pub-sub?
- munchbunny 1y ago
  > There aren't really "listeners" in the sense of a pub-sub? They are still "listeners", it's that events aren't pushed to the listener. Instead the API is designed for sheer throughput.
nasretdinov 1y ago
Well, it's impractical to try to handle messages individually in Kafka, it's designed to acknowledge entire batches (since it's a distributed append-only log). You can still do that, but the performance will be no better than an SQL database

BerislavLopac 1y ago

To be fair, any (immutable) data structure that includes the creation timestamp can be a queue. It might not be a good queue, but it can be used as one.

hamandcheese 1y ago
On this note... has anyone here used object storage directly as a queue? How did it go? You can make a bucket immutable, and entries have timestamps. I don't think any cloud provider makes claims about the accuracy or monoatomicity of these timestamps, so you would merely get an ordering, not necessarily the ordering in which things truly occurred. But I have use cases where that is fine. I believe with a clever naming scheme and cooperating clients it could be made to work.
- dvhh 1y ago
  Once used object storage as queue, you can implement queue semantic at the application level, with one object per entry. But the application was fairly low volume in Data and usage, So eventual consistency and capacity was not an issue. And yes timestamp monotonicity is not guaranteed when multiple client upload at the same time so unique id was given to each client at startup and used for to add guarantee of entries name. Metadata and prefix were used to indicate state of object during processing. Not ideal, but it was cheaper that a DB or a dedicated MQ. The application did not last, but would try again the approach if adapted to stuation.
  - hamandcheese 1y ago
    The application I'm interested in is a log-based artifact registry. Volume would be very low. Much more important is the immutability and durability of the log. I was thinking that writes could be indexed/prefixed into timestamp buckets according to the clients local time. This can't be trusted, of course. But the application consumers could detect and reject any writes whose upload timestamp exceeds a fixed delta from the timestamp bucket it was uploaded to. That allows for arbitrary seeking to any point on the log.

saberience 1y ago

Yes but the practical reality of it is it can be used exactly the same way as you would do a queue and you can make it work just as well as any MQ based system. I know this as I moved from a RabbitMQ system to Kafka for additionally scalability requirements and it worked perfectly. So sure "technically" it's not a queue, but in reality its used as a queue for 1000s of companies around the world for huge production workloads which no MQ system can support.

dxxvi 1y ago
Is it true that a message from a queue will disappear after it is consumed successfully? If yes, at this moment, how do you make kafka topics work as queues?
- deleted 1y ago
  [deleted]
- saberience 1y ago
  Think about this for a second. Kafka offsets are a thing, consumer groups are a thing. It's trivial to ensure that only one message is delivered to only one consumer if that's what you want. Consumer groups track their offset and then commit the offset, the message stays in Kafka but it won't be read again. This IMO is better behaviour than RabbitMQ since you can always re-read messages once they have been processed, whereas generally with MQ systems the message is then marked for deletion and asynchronously deleted.
  - throwwgisgreat 1y ago
    > better behaviour than RabbitMQ since you can always re-read messages once they have been processed I can imagine, a 1 Billion dollar transaction accidentally gets processed by ten thousand client nodes due to a client app synchronization bug, company rethinks its dumb data dumper server strategy...news at 11.
  - otterley 1y ago
    > It's trivial to ensure that only one message is delivered to only one consumer Exactly-once delivery is one of the hardest distributed systems problems. If you’ve “trivially” solved it, please show us your solution.

munchbunny 1y ago

Yup, that's a fair and important distinction, Kafka comes with very little of the ergonomics that make MQ good at what it does. In exchange you get blistering scale, for when blistering scale is the main problem.