10 ms·

I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple language

I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple languages and decent built-in observability) for like... 3 years. Every 6 months I'll check in and see if someone has built it yet, evaluate the alternatives, and come away disappointed. One important feature request that probably would block our adoption: one reason why I prefer a postgres-backed queue over eg. Redis is just to simplify our infra by having fewer servers and technologies in the stack. Adding in RabbitMQ is definitely an extra dependency I'd really like to avoid. (Currently we've settled on graphile-worker which is fine for what it does, but leaves a lot of boxes unchecked.)

abelanger 3y ago
Thank you, appreciate the kind words! What boxes are you looking to check? Yes, I'm not a fan of the RabbitMQ dependency either - see here for the reasoning: https://news.ycombinator.com/item?id=39643940 https://news.ycombinator.com/item?id=39643940. It would take some work to replace this with listen/notify in Postgres, less work to replace this with an in-memory component, but we can't provide the same guarantees in that case.
- jaggederest 3y ago
  I come to this only as an interested observer, but my experience with listen/notify is that it outperforms rabbitmq/kafka in small to medium operations and has always pleasantly surprised me. You might find out it's a little easier than you think to slim your dependency stack down.
  - hosh 3y ago
    How do you handle things when no listeners are available to be notified?
    - abelanger 3y ago
      Presumably there'd be a messages table that you listen/notify on, and you'd replay messages that weren't consumed when a listener rejoins. But yeah, this is the overhead I was referencing.
      - jaggederest 3y ago
        Yep, but practically speaking, you need those records anyway even if you're using another queue to actually distribute the jobs. At least every system I've ever built of a reasonable size has a job audit table anyway. Plus it's an "Enterprise Feature™" so you can differentiate on it if you like that kind of feature-based pricing
        hosh 3y ago
        Postgres's LISTEN/NOTIFY doesn't keep those kinds of records. The whole point of using SKIP LOCK is so that you can make updates to rows to keep those kinds of messages with concurrent consumers.
        jaggederest 3y ago
        Yes. I'm saying you'll manually need to insert some kind of job audit log into a different table. Cheers
      - hosh 3y ago
        With the way LISTEN/NOTIFY works, Postgres doesn't keep a record of messages that are not sent. So you cannot replay this. Unless you know something about postgresql that I don't know about.
        yencabulator 3y ago
        You insert work-to-be-performed into a table, and use NOTIFY only to wake up consumers that there is more work to be had. Consumers that weren't there at the time of NOTIFY can look at the rows in the table at startup.
- kcorbitt 3y ago
  Boxes-wise, I'd like a management interface at least as good as the one Sidekiq had in Rails for years. Would also need some hard numbers around performance and probably a bit more battle-testing before using this in our current product.
BenjieGillam 3y ago
Not sure if you saw it but Graphile Worker supports jobs written in arbitrary languages so long as your OS can execute them: https://worker.graphile.org/docs/tasks#loading-executable-files https://worker.graphile.org/docs/tasks#loading-executable-fi... Would be interested to know what features you feel it’s lacking.
- kcorbitt 3y ago
  That's interesting! Would that still involve each worker node needing to have Nodejs installed to run the process that actually reads from the queue? That's doable, but makes the deployment story a little more annoying/complicated if I want a worker that just runs Python or Rust or something. Feature-wise, the biggest missing pieces from Graphile Worker for me are (1) a robust management web ui and (2) really strong documentation.
  - BenjieGillam 3y ago
    Yes, currently Node is the runtime but we could bundle that up into a binary blob if that would help; one thing to download rather than installing Node and all its dependencies? A UI is a common request, something I’ve been considering investing effort into. I don’t think we’ll ever have one in the core package, but probably as a separate package/plugin (even a third party one); we’ve been thinking more about the events and APIs such a system would need and making these available, and adding a plugin system to enable tighter integration. Could you expand on what’s missing in the documentation? That’s been a focus recently (as you may have noticed with the new expanded docusaurus site linked previously rather than just a README), but documentation can always be improved.
doctorpangloss 3y ago
Why does the RabbitMQ dependency matter? It was pretty painless for me to set up and write tests against. The operator works well and is really simple if you want to save money. I mean, isn’t Hatchett another dependency? Graphile Worker? I like all these things, but why draw the line at one thing over another over essentially aesthetics? You better start believing in dependencies if you’re a programmer.
- eska 3y ago
  Introducing another piece of software instead of using one you already use anyway introduces new failures. That’s hardly aesthetics. As a professional I’m allergic to statements like “you better start believing in X”. How can you even have objective discourse at work like that?
  - doctorpangloss 3y ago
    > Introducing another piece of software instead of using one you already use anyway introduces new failures. Okay, but we're talking about this on a post about using another piece of software. What is the rational for, well this additional dependency, Hatchet, that's okay, and its inevitable failures are okay, but this other dependency, RabbitMQ, which does something different, but will have fewer failures for some objective reasons, that's not okay? Hatchet is very much about aesthetics. What else does Hatchet have going on? It doesn't have a lot of history, it's going to have a lot of bugs. It works as a DSL written in Python annotations, which is very much an aesthetic choice, very much something I see a bunch of AI startups doing, which I personally think is kind of dumb. Like OpenAI tools are "just" JSON schemas, they don't reinvent everything, and yet Trigger, Hatchet, Runloop, etc., they're all doing DSLs. It hews to a specific promotional playbook that is also very aesthetic. Is this not the "objective discourse at work" you are looking for? I am not saying it is bad, I am saying that 99% of people adopting it will be doing so for essentially aesthetic reasons - and being less knowledgable about alternatives might describe 50-80% of the audience, but to me, being less knowledgeable as a "professional" is an aesthetic choice. There's nothing wrong with this. You can get into the weeds about what you meant by whatever you said. I am aware. But I am really saying, I'm dubious of anyone promoting "Use my new thing X which is good because it doesn't introduce a new dependency." It's an oxymoron plainly on its face. It's not in their marketing copy but the author is talking about it here, and maybe the author isn't completely sincere, maybe the author doesn't care and will happily write everything on top of RabbitMQ if someone were willing to pay for it, because that decision doesn't really matter. The author is just being reactive to people's aesthetics, that programmers on social media "like" Postgres more than RabbitMQ, for reasons, and that means you can "only" use one, but that none of those reasons are particularly well informed by experience or whatever, yet nonetheless strongly held. When you want to explain something that doesn't make objective sense when read literally, okay, it might have an aesthetic explanation that makes more sense.
ako 3y ago
Funny how this is vision now. I started my career 29 years ago at a company that build exactly this, but based on oracle. The agents would run on Solaris, aix, vax vms, hpux, windows nt, iris, etc. Was also used to create an automated cicd pipeline to build all binaries on all these different systems.
- throwawaymaths 3y ago
  Also basically has existed as an open source (pro version has web dashboard and complex task zoo) drop-in library (no sidecar dependencies outside of postgres) in Elixir for years called Oban.
  - cpursley 3y ago
    Yep, it feels like half the show hn launches is for infrastructure tooling that already exist natively or as plug and play libraries for Elixir/Erlang. I really try to suggest people skip Node and learn a proper backend language with a solid framework with a proven architecture.
    - zepolen 3y ago
      Oban looks great, how would one run a python cuda based workload on it?
      - hosh 3y ago
        You could shell out to execute with porcelain, make the python a long-running process and use ports, or port your python code to NX.
- sixdimensional 3y ago
  Because people don’t know what they don’t know, and, learning from others (along with human knowledge sharing and transfer) doesn’t seem to be what society often prioritizes in general. Not so much talking about the original post, I think it’s awesome what they are building, and clearly they have learned by observing other things.
bevekspldnw 3y ago
You can do a fair amount of this with Postgres using locks out of the box. It’s not super intuitive but I’ve been using just Postgres and locks in production for many years for large task distribution across independent nodes.
- renegade-otter 3y ago
  I wrote about one simple implementation: https://renegadeotter.com/2023/11/30/job-queues-with-postrgres.html https://renegadeotter.com/2023/11/30/job-queues-with-postrgr...
  - bevekspldnw 3y ago
    Looks very similar to my solution. :-)
deleted 3y ago
[deleted]
rubenfiszel 3y ago
Windmill is is built exactly like that, what box is left unchecked for it if you had time to review it?
- yencabulator 3y ago
  Note that Hatchet is MIT license and Windmill is AGPL-3.. that's enough of a reason for many.
simplyinfinity 3y ago
Hope im not misunderstanding, but have you checked gearman? While I haven't used it personally, ive used similar thing but in c#, namely hangfire.
magic_hamster 3y ago
For what it's worth, RabbitMQ is extremely low maintenance, fire and forget. In the multiple years we've used it in production I can't remember a single time we had an issue with rabbit or that we needed to do anything after the initial set up.

abelanger 3y ago

Thank you, appreciate the kind words! What boxes are you looking to check? Yes, I'm not a fan of the RabbitMQ dependency either - see here for the reasoning: https://news.ycombinator.com/item?id=39643940 https://news.ycombinator.com/item?id=39643940. It would take some work to replace this with listen/notify in Postgres, less work to replace this with an in-memory component, but we can't provide the same guarantees in that case.

jaggederest 3y ago
I come to this only as an interested observer, but my experience with listen/notify is that it outperforms rabbitmq/kafka in small to medium operations and has always pleasantly surprised me. You might find out it's a little easier than you think to slim your dependency stack down.
- hosh 3y ago
  How do you handle things when no listeners are available to be notified?
  - abelanger 3y ago
    Presumably there'd be a messages table that you listen/notify on, and you'd replay messages that weren't consumed when a listener rejoins. But yeah, this is the overhead I was referencing.
    - jaggederest 3y ago
      Yep, but practically speaking, you need those records anyway even if you're using another queue to actually distribute the jobs. At least every system I've ever built of a reasonable size has a job audit table anyway. Plus it's an "Enterprise Feature™" so you can differentiate on it if you like that kind of feature-based pricing
      - hosh 3y ago
        Postgres's LISTEN/NOTIFY doesn't keep those kinds of records. The whole point of using SKIP LOCK is so that you can make updates to rows to keep those kinds of messages with concurrent consumers.
        jaggederest 3y ago
        Yes. I'm saying you'll manually need to insert some kind of job audit log into a different table. Cheers
    - hosh 3y ago
      With the way LISTEN/NOTIFY works, Postgres doesn't keep a record of messages that are not sent. So you cannot replay this. Unless you know something about postgresql that I don't know about.
      - yencabulator 3y ago
        You insert work-to-be-performed into a table, and use NOTIFY only to wake up consumers that there is more work to be had. Consumers that weren't there at the time of NOTIFY can look at the rows in the table at startup.
kcorbitt 3y ago
Boxes-wise, I'd like a management interface at least as good as the one Sidekiq had in Rails for years. Would also need some hard numbers around performance and probably a bit more battle-testing before using this in our current product.

BenjieGillam 3y ago

Not sure if you saw it but Graphile Worker supports jobs written in arbitrary languages so long as your OS can execute them: https://worker.graphile.org/docs/tasks#loading-executable-files https://worker.graphile.org/docs/tasks#loading-executable-fi... Would be interested to know what features you feel it’s lacking.

kcorbitt 3y ago
That's interesting! Would that still involve each worker node needing to have Nodejs installed to run the process that actually reads from the queue? That's doable, but makes the deployment story a little more annoying/complicated if I want a worker that just runs Python or Rust or something. Feature-wise, the biggest missing pieces from Graphile Worker for me are (1) a robust management web ui and (2) really strong documentation.
- BenjieGillam 3y ago
  Yes, currently Node is the runtime but we could bundle that up into a binary blob if that would help; one thing to download rather than installing Node and all its dependencies? A UI is a common request, something I’ve been considering investing effort into. I don’t think we’ll ever have one in the core package, but probably as a separate package/plugin (even a third party one); we’ve been thinking more about the events and APIs such a system would need and making these available, and adding a plugin system to enable tighter integration. Could you expand on what’s missing in the documentation? That’s been a focus recently (as you may have noticed with the new expanded docusaurus site linked previously rather than just a README), but documentation can always be improved.

doctorpangloss 3y ago

Why does the RabbitMQ dependency matter? It was pretty painless for me to set up and write tests against. The operator works well and is really simple if you want to save money. I mean, isn’t Hatchett another dependency? Graphile Worker? I like all these things, but why draw the line at one thing over another over essentially aesthetics? You better start believing in dependencies if you’re a programmer.

eska 3y ago
Introducing another piece of software instead of using one you already use anyway introduces new failures. That’s hardly aesthetics. As a professional I’m allergic to statements like “you better start believing in X”. How can you even have objective discourse at work like that?
- doctorpangloss 3y ago
  > Introducing another piece of software instead of using one you already use anyway introduces new failures. Okay, but we're talking about this on a post about using another piece of software. What is the rational for, well this additional dependency, Hatchet, that's okay, and its inevitable failures are okay, but this other dependency, RabbitMQ, which does something different, but will have fewer failures for some objective reasons, that's not okay? Hatchet is very much about aesthetics. What else does Hatchet have going on? It doesn't have a lot of history, it's going to have a lot of bugs. It works as a DSL written in Python annotations, which is very much an aesthetic choice, very much something I see a bunch of AI startups doing, which I personally think is kind of dumb. Like OpenAI tools are "just" JSON schemas, they don't reinvent everything, and yet Trigger, Hatchet, Runloop, etc., they're all doing DSLs. It hews to a specific promotional playbook that is also very aesthetic. Is this not the "objective discourse at work" you are looking for? I am not saying it is bad, I am saying that 99% of people adopting it will be doing so for essentially aesthetic reasons - and being less knowledgable about alternatives might describe 50-80% of the audience, but to me, being less knowledgeable as a "professional" is an aesthetic choice. There's nothing wrong with this. You can get into the weeds about what you meant by whatever you said. I am aware. But I am really saying, I'm dubious of anyone promoting "Use my new thing X which is good because it doesn't introduce a new dependency." It's an oxymoron plainly on its face. It's not in their marketing copy but the author is talking about it here, and maybe the author isn't completely sincere, maybe the author doesn't care and will happily write everything on top of RabbitMQ if someone were willing to pay for it, because that decision doesn't really matter. The author is just being reactive to people's aesthetics, that programmers on social media "like" Postgres more than RabbitMQ, for reasons, and that means you can "only" use one, but that none of those reasons are particularly well informed by experience or whatever, yet nonetheless strongly held. When you want to explain something that doesn't make objective sense when read literally, okay, it might have an aesthetic explanation that makes more sense.

ako 3y ago

Funny how this is vision now. I started my career 29 years ago at a company that build exactly this, but based on oracle. The agents would run on Solaris, aix, vax vms, hpux, windows nt, iris, etc. Was also used to create an automated cicd pipeline to build all binaries on all these different systems.

throwawaymaths 3y ago
Also basically has existed as an open source (pro version has web dashboard and complex task zoo) drop-in library (no sidecar dependencies outside of postgres) in Elixir for years called Oban.
- cpursley 3y ago
  Yep, it feels like half the show hn launches is for infrastructure tooling that already exist natively or as plug and play libraries for Elixir/Erlang. I really try to suggest people skip Node and learn a proper backend language with a solid framework with a proven architecture.
  - zepolen 3y ago
    Oban looks great, how would one run a python cuda based workload on it?
    - hosh 3y ago
      You could shell out to execute with porcelain, make the python a long-running process and use ports, or port your python code to NX.
sixdimensional 3y ago
Because people don’t know what they don’t know, and, learning from others (along with human knowledge sharing and transfer) doesn’t seem to be what society often prioritizes in general. Not so much talking about the original post, I think it’s awesome what they are building, and clearly they have learned by observing other things.

bevekspldnw 3y ago

You can do a fair amount of this with Postgres using locks out of the box. It’s not super intuitive but I’ve been using just Postgres and locks in production for many years for large task distribution across independent nodes.

renegade-otter 3y ago
I wrote about one simple implementation: https://renegadeotter.com/2023/11/30/job-queues-with-postrgres.html https://renegadeotter.com/2023/11/30/job-queues-with-postrgr...
- bevekspldnw 3y ago
  Looks very similar to my solution. :-)

deleted 3y ago

[deleted]

rubenfiszel 3y ago

Windmill is is built exactly like that, what box is left unchecked for it if you had time to review it?

yencabulator 3y ago
Note that Hatchet is MIT license and Windmill is AGPL-3.. that's enough of a reason for many.

simplyinfinity 3y ago

Hope im not misunderstanding, but have you checked gearman? While I haven't used it personally, ive used similar thing but in c#, namely hangfire.

magic_hamster 3y ago

For what it's worth, RabbitMQ is extremely low maintenance, fire and forget. In the multiple years we've used it in production I can't remember a single time we had an issue with rabbit or that we needed to do anything after the initial set up.