3 ms·

We’ve implemented two types of queues using Postgres tables in production and it’s been working nicely. Happy to answer questions about our usage if it’s useful

• kumarvvr 5y ago
  How scalable do you think your system is? Can it handle 10k req/sec? My usage is much lower than that but I want to target that number.
  • bfelbo 5y ago
    We're currently keeping it as simple as possible. We just use a table and this query that runs every second: "DELETE FROM job_queue RETURNING type, info;" Works great and reliably so far. I think our current volume is on the order of 100-1000 req/sec. We'll likely switch to graphile-worker when we need more performance, but we're all about avoiding premature optimization. That library has been benchmarked to handle 10k req/sec: https://github.com/graphile/worker#performance https://github.com/graphile/worker#performance
  • pritambarhate 5y ago
    PgSQL table based job queues should be able to handle that kind of load, but to support it you will need very high IOPS based disk. If you have NVME SSD paired with decent CPU, it should be able to manage this. This is mainly because PgSQL based queue is durable by default and based on how you implement your inserts, every new message will need a disk based commit.
• kumarvvr 5y ago
  Would also love to hear or read about your implementation, the general architecture and failsafe constructs. Thank you.