3 ms·

Good article! I am currently implementing a project in which we use Postgres to track job state (eg., run status, failures, timings, resource usage, related lo

Good article! I am currently implementing a project in which we use Postgres to track job state (eg., run status, failures, timings, resource usage, related log entries), but Kafka as the actual queueing mechanism -- thus bypassing the challenges mentioned in the article but still getting the best of Postgres. This way we have complete, introspectable, measurable history about every queue item. It greatly simplifies the Postgres part of it (state updates are always appends, no locking) and thanks to Kafka, increases performance and scalability. It also adds a measure of safety: We can detect "lost" jobs that disappear because of data failure, bugs in Kafka, failing clients etc. We know that if a job was never logged as "complete", it probably died. The job log also functions as an audit log, and we also intend to use it for certain jobs that benefit from being incremental and from able to continue from when it last left (for example, feed processing).

brandur 11y ago
Interesting approach! I'd be curious to hear about the mechanic that you came up with for division of labor among workers — since every client is essentially reading the same stream, I guess you'd have to distribute jobs based on job_id modulo worker_number or something like that?
- atombender 11y ago
  I suppose one could use a round-robin sharding approach like you mention, but it goes against Kafka's design, and it's not necessary. Kafka divides a queue into partitions. Each partition is a completely independent silo. When you publish messages, Kafka distributes them across partititons. When you read messages, you always read from a partition. This means partitions are also the unit of parallelism: You don't want multiple workers on a single partition (because of the labour division problem you mention). Rather, Kafka expects you to have one partition per worker. This is more elegant than it sounds if you're coming from something like RabbitMQ. Partitions (ie., queues) in Kafka are append-only and strictly linear; unlike RabbitMQ, you can never "nack" a message in a way that results in the message ending up at the back of the queue and thus violating the original message order. Rather, Kafka expects each consumer to maintain its "read position" in the queue. Failure handling, then, is simply a matter of winding back the read position. And unlike RabbitMQ, there's less need for complicated routing, dead-letter exchanges and so on, because rather than move messages around, you're just moving a cursor. Of course, message order is only preserved within a single partition; if you publish messages A, B and C and you have 3 partitions and 3 workers, then in a real world, messages may be processed in the order C, B, A. That sounds bad, but then other queue solutions such as Que or RabbitMQ suffer from the exact same problem: If you run 3 workers against one queue, your queue may supply each worker with messages in the right order, but there's no guarantee that they will be processed in that order. The only way to guarantee ordering is to have just one worker per queue, using some kind of locking (RabbitMQ does support "exclusive" consumers). But then you don't get any parallelism at all. So I think Kafka's solution is quite sane, even if it's more low-level and less developer-friendly than AMQP.

brandur 11y ago

Interesting approach! I'd be curious to hear about the mechanic that you came up with for division of labor among workers — since every client is essentially reading the same stream, I guess you'd have to distribute jobs based on job_id modulo worker_number or something like that?

atombender 11y ago
I suppose one could use a round-robin sharding approach like you mention, but it goes against Kafka's design, and it's not necessary. Kafka divides a queue into partitions. Each partition is a completely independent silo. When you publish messages, Kafka distributes them across partititons. When you read messages, you always read from a partition. This means partitions are also the unit of parallelism: You don't want multiple workers on a single partition (because of the labour division problem you mention). Rather, Kafka expects you to have one partition per worker. This is more elegant than it sounds if you're coming from something like RabbitMQ. Partitions (ie., queues) in Kafka are append-only and strictly linear; unlike RabbitMQ, you can never "nack" a message in a way that results in the message ending up at the back of the queue and thus violating the original message order. Rather, Kafka expects each consumer to maintain its "read position" in the queue. Failure handling, then, is simply a matter of winding back the read position. And unlike RabbitMQ, there's less need for complicated routing, dead-letter exchanges and so on, because rather than move messages around, you're just moving a cursor. Of course, message order is only preserved within a single partition; if you publish messages A, B and C and you have 3 partitions and 3 workers, then in a real world, messages may be processed in the order C, B, A. That sounds bad, but then other queue solutions such as Que or RabbitMQ suffer from the exact same problem: If you run 3 workers against one queue, your queue may supply each worker with messages in the right order, but there's no guarantee that they will be processed in that order. The only way to guarantee ordering is to have just one worker per queue, using some kind of locking (RabbitMQ does support "exclusive" consumers). But then you don't get any parallelism at all. So I think Kafka's solution is quite sane, even if it's more low-level and less developer-friendly than AMQP.