5 ms·

Introducing PGMQ: Simple Message Queues Built on Postgres

10 points by chuckhend 3y ago

ants_a 3y ago
Is there a reason why this is implemented in Rust instead of plpgsql? At first glance I didn't see it using any features that would need a loadable extension. Plain SQL based solution would make using it on managed database services much easier.
- chuckhend 3y ago
  It probably could be implemented as pure plpgsql for some of its features (basic send/receive). The long-term plan is to add additional message type support (protobuf, msgpack, etc), embed a metrics exporter (prometheus?), push notifications, and. for those we want to pull in Rust modules from open source. I bet the project could be refactored to expose the parts you're looking for as plpgsql though. Distribution of Postgres extensions is a challenging problem that we're working on over in https://github.com/tembo-io/trunk https://github.com/tembo-io/trunk // https://pgt.dev/ https://pgt.dev/. One of the goals there is to make it easier to install extensions.
stephane-klein 3y ago
On the same subject (job queue based on PostgreSQL), I'm successfully using the https://github.com/graphile/worker/ https://github.com/graphile/worker/ (NodeJS) project in production. Jobs are written in Javascript.
samaysharma 3y ago
We're excited to share an implementation of Message Queues as a Postgres Extension (which we've been using in production), following best practices recommended by others around this topic. Please share any suggestions and feedback.
cauchyk 3y ago
At my previous job there were multiple use-cases where we used RabbitMQ in conjunction with Postgres. This would've made life so much easier. Are there any instructions on installing? It'd be great if the `.so`s are released as part of GH releases.
- chuckhend 3y ago
  If you're on Linux and pg15, you can install it with https://github.com/tembo-io/trunk https://github.com/tembo-io/trunk. It is only tested on pg15 but we are actively working to increase OS and Postgres version support. `trunk install pgmq` should install with the pg_config on your path by default. If this does not work you can build from source. There is a guide in the contributing docs https://github.com/tembo-io/pgmq/blob/main/CONTRIBUTING.md https://github.com/tembo-io/pgmq/blob/main/CONTRIBUTING.md
pabloem1234 3y ago
This is interesting. I'd love to see benchmarks to have a sense of what to expect.
- chuckhend 3y ago
  Awesome! We'll blogging about its benchmarks compared to Redis and SQS soon.

10 points by chuckhend 3y ago

Is there a reason why this is implemented in Rust instead of plpgsql? At first glance I didn't see it using any features that would need a loadable extension. Plain SQL based solution would make using it on managed database services much easier.

chuckhend 3y ago
It probably could be implemented as pure plpgsql for some of its features (basic send/receive). The long-term plan is to add additional message type support (protobuf, msgpack, etc), embed a metrics exporter (prometheus?), push notifications, and. for those we want to pull in Rust modules from open source. I bet the project could be refactored to expose the parts you're looking for as plpgsql though. Distribution of Postgres extensions is a challenging problem that we're working on over in https://github.com/tembo-io/trunk https://github.com/tembo-io/trunk // https://pgt.dev/ https://pgt.dev/. One of the goals there is to make it easier to install extensions.

stephane-klein 3y ago

On the same subject (job queue based on PostgreSQL), I'm successfully using the https://github.com/graphile/worker/ https://github.com/graphile/worker/ (NodeJS) project in production. Jobs are written in Javascript.

samaysharma 3y ago

We're excited to share an implementation of Message Queues as a Postgres Extension (which we've been using in production), following best practices recommended by others around this topic. Please share any suggestions and feedback.

cauchyk 3y ago

At my previous job there were multiple use-cases where we used RabbitMQ in conjunction with Postgres. This would've made life so much easier. Are there any instructions on installing? It'd be great if the `.so`s are released as part of GH releases.

chuckhend 3y ago
If you're on Linux and pg15, you can install it with https://github.com/tembo-io/trunk https://github.com/tembo-io/trunk. It is only tested on pg15 but we are actively working to increase OS and Postgres version support. `trunk install pgmq` should install with the pg_config on your path by default. If this does not work you can build from source. There is a guide in the contributing docs https://github.com/tembo-io/pgmq/blob/main/CONTRIBUTING.md https://github.com/tembo-io/pgmq/blob/main/CONTRIBUTING.md

pabloem1234 3y ago

This is interesting. I'd love to see benchmarks to have a sense of what to expect.

chuckhend 3y ago
Awesome! We'll blogging about its benchmarks compared to Redis and SQS soon.