Sure, that's actually very similar to the the system we built internally. But I don't see how Kafka as a messaging layer helps in your system, either. Presumabl

Sure, that's actually very similar to the the system we built internally. But I don't see how Kafka as a messaging layer helps in your system, either. Presumably, the consumers won't advance their offset in their partition until the job is done, because they are not ready to start a new job. So you will still get the same blocking behavior. In addition, I don't see how you implement priority in a system like this, since Kafka partitions are append-only. One has to process the former start_job messages before you reach the higher priority items. Having a broker hand out items in priority to workers makes a lot more sense to me. The job system you described is definitely something missing from the OSS world, and we'd love to open up our system for this eventually. While Disque is not that piece, I think it makes more sense than Kafka.