4 ms·
Kafka is not a queue. Kafka's parallelism is limited by the number of partitions you allocate, and you have to be sure to avoid head of line blocking. Not the
by staticassertion 5y ago
Kafka is not a queue. Kafka's parallelism is limited by the number of partitions you allocate, and you have to be sure to avoid head of line blocking.
Not the case with a queue.
- jpgvm 5y agoExactly. If you do want something very scalable that fixes these problems but shares a lot of architectural similarity with Kafka then you should check out Apache Pulsar.
- Spivak 5y agoThis needs to be sung from the rooftops every time Kafka is mentioned. It's an amazing tool but it is the wrong wrong wrong tool if you need a queue. It will bite you in the ass and you'll be left with someone breathing down your neck wondering why jobs are processing so slowly and why you can't just spin up more workers.
- fnord77 5y agowhat is "head of line" blocking?
- staticassertion 5y agoA single partition is intended to be processed, more or less, in by a single worker. If one of those messages, for whatever reason, ends up being really expensive, or flaky, you can't move on until you've handled it. That's head of line blocking.