4 ms·
Thank you, appreciate the kind words! What boxes are you looking to check? Yes, I'm not a fan of the RabbitMQ dependency either - see here for the reasoning: h
by abelanger 3y ago
Thank you, appreciate the kind words! What boxes are you looking to check?
Yes, I'm not a fan of the RabbitMQ dependency either - see here for the reasoning: https://news.ycombinator.com/item?id=39643940 https://news.ycombinator.com/item?id=39643940.
It would take some work to replace this with listen/notify in Postgres, less work to replace this with an in-memory component, but we can't provide the same guarantees in that case.
- jaggederest 3y agoI come to this only as an interested observer, but my experience with listen/notify is that it outperforms rabbitmq/kafka in small to medium operations and has always pleasantly surprised me. You might find out it's a little easier than you think to slim your dependency stack down.
- hosh 3y agoHow do you handle things when no listeners are available to be notified?
- abelanger 3y agoPresumably there'd be a messages table that you listen/notify on, and you'd replay messages that weren't consumed when a listener rejoins. But yeah, this is the overhead I was referencing.
- jaggederest 3y agoYep, but practically speaking, you need those records anyway even if you're using another queue to actually distribute the jobs. At least every system I've ever built of a reasonable size has a job audit table anyway. Plus it's an "Enterprise Feature™" so you can differentiate on it if you like that kind of feature-based pricing
- hosh 3y agoPostgres's LISTEN/NOTIFY doesn't keep those kinds of records. The whole point of using SKIP LOCK is so that you can make updates to rows to keep those kinds of messages with concurrent consumers.
- jaggederest 3y agoYes. I'm saying you'll manually need to insert some kind of job audit log into a different table. Cheers
- hosh 3y agoWith the way LISTEN/NOTIFY works, Postgres doesn't keep a record of messages that are not sent. So you cannot replay this. Unless you know something about postgresql that I don't know about.
- yencabulator 3y agoYou insert work-to-be-performed into a table, and use NOTIFY only to wake up consumers that there is more work to be had. Consumers that weren't there at the time of NOTIFY can look at the rows in the table at startup.
- hosh 3y agoI see. So the notify is just to say there is work to be performed, but there is no payload that includes the job. The consumer still has to make a query. If there isn’t enough work, the queries should come back empty. This saves from having to poll, but not a true push system.
- jaggederest 3y agoas far as I can tell NOTIFY is fanout, in the sense that it will send a message to all the LISTENing connections, so it wouldn't make sense in that context anyway. It's not one-to-one, it's about making sure that jobs get picked up in a timely fashion. If you're doing something fancier with event sourcing or equivalent, you can send events via NOTIFY, and have clients decide what to do with those events then. Quoth the manual: "The NOTIFY command sends a notification event together with an optional “payload” string to each client application that has previously executed LISTEN channel for the specified channel name in the current database. Notifications are visible to all users."
- hosh 3y agoNotify can be triggered with stored procedures to send payloads related to changes to a table. It can be set up to send the id of a row that was inserted or updated, for example. (But WAL replication is usually better for this)
- kcorbitt 3y agoBoxes-wise, I'd like a management interface at least as good as the one Sidekiq had in Rails for years. Would also need some hard numbers around performance and probably a bit more battle-testing before using this in our current product.