3 ms·

> Use Postgres as a message queue with SKIP LOCKED instead of Kafka (if you only need a message queue). I wouldn't describe Kafka as a message queue. It's more

> Use Postgres as a message queue with SKIP LOCKED instead of Kafka (if you only need a message queue). I wouldn't describe Kafka as a message queue. It's more of a distributed commit log.

pdntspa 4y ago
I have done this, and the one thing I would note is that if you are requesting a maximum number of rows per call, and if something in the queue breaks for all returned rows (like because it crashed with an exception or something and its status field isnt updated), then you can run a situation where every call to get more rows goes to the same broken rows and the queue becomes blocked. I hadn't figured out a great solution for this but otherwise my SKIP LOCKED queue worked great!
KingOfCoders 4y ago
Yes I do agree. But looking into many startups, many of them use it as a message queue with the ability for transformations (E.g. Spark).
- jpgvm 4y ago
  It's fine for streaming, i.e Spark and friends. I generally define streaming as the success of each individual message is entirely uniform, i.e the system can either process messages or it cannot. If this holds none of Kafkas drawbacks should significantly affect you. When people talk messaging they normally mean something lower latency, potentially out of order and where the success of each message is likely independent of other messages. For these use-cases Kafka is not well suited. For these cases you will want something like Pulsar, SQS, PubSub, etc.
manicennui 4y ago
Unfortunately many (most?) people use Kafka as a message queue with larger message size.
ARandomerDude 4y ago
Messaging is the first use-case given by the Kafka docs [1] and is the first core capability they mention on the Kafka landing page [2]. 1. https://kafka.apache.org/documentation/#uses_messaging https://kafka.apache.org/documentation/#uses_messaging 2. https://kafka.apache.org/ https://kafka.apache.org/ I use (and think of) Kafka as a message bus, with distributed commit logging being the mechanism for how it accomplishes that task.
- jpgvm 4y ago
  It's really not. It's a distributed log. Because it's really a log and it's partitioned it has a whole host of issues that make it a very poor messaging system like hot partions, head of line blocking etc. If what you want is a proper messaging system where the underlying model is still a distributed log then you should look at Apache Pulsar. It can provide the same semantics as Kafka but can properly implement job queues and messaging semantics ala SQS, Google PubSub, et al.

> Use Postgres as a message queue with SKIP LOCKED instead of Kafka (if you only need a message queue). I wouldn't describe Kafka as a message queue. It's more of a distributed commit log.

pdntspa 4y ago

I have done this, and the one thing I would note is that if you are requesting a maximum number of rows per call, and if something in the queue breaks for all returned rows (like because it crashed with an exception or something and its status field isnt updated), then you can run a situation where every call to get more rows goes to the same broken rows and the queue becomes blocked. I hadn't figured out a great solution for this but otherwise my SKIP LOCKED queue worked great!

KingOfCoders 4y ago

Yes I do agree. But looking into many startups, many of them use it as a message queue with the ability for transformations (E.g. Spark).

jpgvm 4y ago
It's fine for streaming, i.e Spark and friends. I generally define streaming as the success of each individual message is entirely uniform, i.e the system can either process messages or it cannot. If this holds none of Kafkas drawbacks should significantly affect you. When people talk messaging they normally mean something lower latency, potentially out of order and where the success of each message is likely independent of other messages. For these use-cases Kafka is not well suited. For these cases you will want something like Pulsar, SQS, PubSub, etc.

manicennui 4y ago

Unfortunately many (most?) people use Kafka as a message queue with larger message size.

ARandomerDude 4y ago

Messaging is the first use-case given by the Kafka docs [1] and is the first core capability they mention on the Kafka landing page [2]. 1. https://kafka.apache.org/documentation/#uses_messaging https://kafka.apache.org/documentation/#uses_messaging 2. https://kafka.apache.org/ https://kafka.apache.org/ I use (and think of) Kafka as a message bus, with distributed commit logging being the mechanism for how it accomplishes that task.

jpgvm 4y ago
It's really not. It's a distributed log. Because it's really a log and it's partitioned it has a whole host of issues that make it a very poor messaging system like hot partions, head of line blocking etc. If what you want is a proper messaging system where the underlying model is still a distributed log then you should look at Apache Pulsar. It can provide the same semantics as Kafka but can properly implement job queues and messaging semantics ala SQS, Google PubSub, et al.