2 ms·

We process around 1 million events a day using a queue like this in Postgres, and have processed over 400 million events since the system this is used in went l

We process around 1 million events a day using a queue like this in Postgres, and have processed over 400 million events since the system this is used in went live. Only issue we've had was slow queries due to the table size, as we keep an archive of all the events processed, but some scheduled vacuums every so often kept that under control.

djbusby 3y ago
Active Queue table and then archive jobs to a JobDone table? I do that. Queue table is small but archive goes back many months
- MattIPv4 3y ago
  We just have a single table, with a column indicating if the job has been taken by a worker or not. Probably could get a bit more performance out of it by splitting into two tables, but it works as it is for now.
- paulddraper 3y ago
  In modern PG you can use partitioned table for a similar effect.
andrewstuart 3y ago
Partial indexes might help.
- emilsedgh 3y ago
  Exactly. A partial index should make things fly here.
- runeks 3y ago
  Also: an ordered index that matches the ordering clause in your job-grabbing query. This is useful if you have lots of pending jobs.