15 ms·

Show HN: Hatchet – Open-source distributed task queue

578 points by abelanger 3y ago

Hello HN, we're Gabe and Alexander from Hatchet (https://hatchet.run https://hatchet.run), we're working on an open-source, distributed task queue. It's an alternative to tools like Celery for Python and BullMQ for Node.js, primarily focused on reliability and observability. It uses Postgres for the underlying queue. Why build another managed queue? We wanted to build something with the benefits of full transactional enqueueing - particularly for dependent, DAG-style execution - and felt strongly that Postgres solves for 99.9% of queueing use-cases better than most alternatives (Celery uses Redis or RabbitMQ as a broker, BullMQ uses Redis). Since the introduction of SKIP LOCKED and the milestones of recent PG releases (like active-active replication), it's becoming more feasible to horizontally scale Postgres across multiple regions and vertically scale to 10k TPS or more. Many queues (like BullMQ) are built on Redis and data loss can occur when suffering OOM if you're not careful, and using PG helps avoid an entire class of problems. We also wanted something that was significantly easier to use and debug for application developers. A lot of times the burden of building task observability falls on the infra/platform team (for example, asking the infra team to build a Grafana view for their tasks based on exported prom metrics). We're building this type of observability directly into Hatchet. What do we mean by "distributed"? You can run workers (the instances which run tasks) across multiple VMs, clusters and regions - they are remotely invoked via a long-lived gRPC connection with the Hatchet queue. We've attempted to optimize our latency to get our task start times down to 25-50ms and much more optimization is on the roadmap. We also support a number of extra features that you'd expect, like retries, timeouts, cron schedules, dependent tasks. A few things we're currently working on - we use RabbitMQ (confusing, yes) for pub/sub between engine components and would prefer to just use Postgres, but didn't want to spend additional time on the exchange logic until we built a stable underlying queue. We are also considering the use of NATS for engine-engine and engine-worker connections. We'd greatly appreciate any feedback you have and hope you get the chance to try out Hatchet.

dataangel 3y ago
> Distributed > Built on PostGRES Not what people usually mean by distributed, caveat emptor
- abelanger 3y ago
  Citus Data would like a word.
topicseed 3y ago
What specific strategies does Hatchet employ to guarantee fault tolerance and enable durable execution? How does it handle partial failures in multi-step workflows?
- abelanger 3y ago
  Each task in Hatchet is backed by a workflow [1]. Workflows are predefined steps which are persisted in PostgreSQL. If a worker dies or crashes midway through (stops heartbeating to the engine), we reassign tasks (assuming they have retries left). We also track timeouts in the database, which means if we miss a timeout, we simply retry after some amount of time. Like I mentioned in the post, we avoid some classes of faults just by relying on PostgreSQL and persisting each workflow run, so you don't need to time out with distributed locks in Redis, for example, or worry about data loss if Redis OOMs. Our `ticker` service is basically its own worker which is assigned a lease for each step run. We also store the input/output of each workflow step in the database. So resuming a multi-step workflow is pretty simple - we just replay the step with the same input. To zoom out a bit - unlike many alternatives [2], the execution path of a multi-step workflow in Hatchet is declared ahead of time. There are tradeoffs to this approach; it makes it much easier to run a single-step workflow or if you know the workflow execution path ahead of time. You also avoid classes of problems related to workflow versioning, we can gracefully drain older workflow version with a different execution path. It's also more natural to debug and see a DAG execution instead of debugging procedural logic. The clear tradeoff is that you can't try...catch the execution of a single task or concatenate a bunch of futures that you wait for later. Roadmap-wise, we're considering adding procedural execution on top of our workflows concept. Which means providing a nice API for calling `await workflow.run` and capturing errors. These would be a higher-level concept in Hatchet and are not built yet. There are some interesting concepts around using semaphores and durable leases that are relevant here, which we're exploring [3]. [1] https://docs.hatchet.run/home/basics/workflows https://docs.hatchet.run/home/basics/workflows [2] https://temporal.io https://temporal.io [3] https://www.citusdata.com/blog/2016/08/12/state-machines-to-run-databases/ https://www.citusdata.com/blog/2016/08/12/state-machines-to-...
  - topicseed 3y ago
nextworddev 3y ago
I’m interested in self hosting this. What’s the recommendation here for state persistence and self healing? Wish there was a guide for a small team who wants to self host before trying managed cloud
- abelanger 3y ago
  I think we might have had a dead link in the README to our self-hosting guide, here it is: https://docs.hatchet.run/self-hosting https://docs.hatchet.run/self-hosting. The component which needs the highest uptime is our ingestion service [1]. This ingests events from the Hatchet SDKs and is responsible for writing the workflow execution path, and then sends messages downstream to our other engine components. This is a horizontally scalable service and you should run at least 2 replicas across different AZs. Also see how to configure different services for engine components [2]. The other piece of this is PostgreSQL, use your favorite managed provider which has point-in-time restores and backups. This is the core of our self-healing, I'm not sure where it makes sense to route writes if the primary goes down. Let me know what you need for self-hosted docs, happy to write them up for you. [1] https://github.com/hatchet-dev/hatchet/tree/main/internal/services/ingestor https://github.com/hatchet-dev/hatchet/tree/main/internal/se... [2] https://docs.hatchet.run/self-hosting/configuration-options#services-configuration https://docs.hatchet.run/self-hosting/configuration-options#...
sixhobbits 3y ago
One of my favourite spaces and presentation in readme is clear and immediately told me what it is and most of the key information that I usually complain is missing. However I am still missing a section on why this is different than any of the other existing and more mature solutions. What led you to develop this over existing options and what different tradeoffs did you make? Extra points if you can concisely tell me what you do badly that your 'competitors' do well because I don't believe there is a one best solution in this space, it is all tradeoffs
- sixhobbits 3y ago
  Sorry I am dumb and commented after clicking on the link. I would just add your hn text to the readme as that is exactly what I was looking for
  - abelanger 3y ago
    Done [1]. We'll expand this section over time. There are also definite tradeoffs to our architecture - spoke to someone wanting the equivalent 1.5m PutRecord/s in Kinesis, which we're definitely not ready for because we're persist every event + task execution in Postgres. [1] https://github.com/hatchet-dev/hatchet/blob/main/README.md#how-does-this-compare-to-alternatives-celery-bullmq https://github.com/hatchet-dev/hatchet/blob/main/README.md#h...
tzahifadida 3y ago
Why not use postgres listen/notify instead of rabbitmq pub sub.
- anentropic 3y ago
  It uses Postgres rather than RabbitMQ: https://github.com/hatchet-dev/hatchet?tab=readme-ov-file#how-does-this-compare-to-alternatives-celery-bullmq https://github.com/hatchet-dev/hatchet?tab=readme-ov-file#ho...
  - anentropic 3y ago
    I see... apparently it uses both
- abelanger 3y ago
  When I started on this codebase, we needed to implement some custom exchange logic that maps very neatly to fanout exchanges and non-durable queues in RabbitMQ and weren't built out on our PostgreSQL layer yet. This was a bootstrapping problem. Like I mentioned in the comment, we'd like to switch to pub/sub pattern that lets us distribute our engine over multiple geographies. Listen/notify could be the answer once we migrate to PG 16, though there are some concerns around connection poolers like pg_bouncer having limited support for listen/notify. There's a Github discussion on this if you're curious: https://github.com/hatchet-dev/hatchet/discussions/224 https://github.com/hatchet-dev/hatchet/discussions/224.
  - tzahifadida 3y ago
    I use haproxy with go listen notify of one of the libs. It works as long as the connection is up. I.e.i have a timeout of 30 min configured in haproxy. Then you have to assume you lost sync and recheck. That is not that bad every 30min... at least for me. You can configure to never close...
Kluggy 3y ago
In https://docs.hatchet.run/home/quickstart/installation https://docs.hatchet.run/home/quickstart/installation, it says > Welcome to Hatchet! This guide walks you through getting set up on Hatchet Cloud. If you'd like to self-host Hatchet, please see the self-hosted quickstart instead. but the link to "self-hosted quickstart" links back to the same page
- abelanger 3y ago
  This should be fixed now, here's the direct link: https://docs.hatchet.run/self-hosting https://docs.hatchet.run/self-hosting.
kevinlu1248 3y ago
We're building a webhook services on FastAPI + Celery + Redis + Grafana + Loki and the experience with setting up every service incrementally was miserable, and even then it feels like logs are being dropped and we run into reliability issues. Felt like something like this should exist already but I couldn't find anything at the time. Really excited to see where this takes us!
- tasn 3y ago
  That's exactly why we built Svix[1]. Building webhooks services, even with amazing tools like FastAPI, Celery and Redis is still a big pain. So we just built a product to solve it. Hatchet looks cool nonetheless. Queues are a pain for many other use-cases too. 1: https://www.svix.com https://www.svix.com
pyrossh 3y ago
How is this different from pg-boss[1]? Other than the distributed part it also seems to use skip locked. [1] https://github.com/timgit/pg-boss https://github.com/timgit/pg-boss
- abelanger 3y ago
  I haven't used pg-boss, and feature-wise it looks very similar and is an impressive project. The core difference is that pg-boss is a library while Hatchet is a separate service which runs independently of your workers. This service also provides a UI and API for interacting with Hatchet - I don't think pg-boss has those things, so you'd probably have to build out observability yourself. This doesn't make a huge difference when you're at 1 worker, but having each worker poll your database can lead to DB issues if you're not careful - I've seen some pretty low-throughput setups for very long-running jobs using a database with 60 CPUs because of polling workers. Hatchet distributes in two layers - the "engine" and the "worker" layer. Each engine polls the database and fans out to the workers over a long-lived gRPC connection. This reduces pressure on the DB and lets us manage which workers to assign tasks to based on things like max concurrent runs on each worker or worker health.
radus 3y ago
You've explained your value proposition vs. celery, but I'm curious if you also see Hatchet as an alternative to Nextflow/Snakemake which are commonly used in bioinformatics.
bluehadoop 3y ago
How does this compare against Temporal/Cadence/Conductor? Does hatchet also support durable execution? https://temporal.io/ https://temporal.io/ https://cadenceworkflow.io/ https://cadenceworkflow.io/ https://conductor-oss.org/ https://conductor-oss.org/
- abelanger 3y ago
  It's very similar - I used Temporal at a previous company to run a couple million workflows per month. The gRPC networking with workers is the most similar component, I especially liked that I only had to worry about an http2 connection with mTLS instead of a different broker protocol. Temporal is a powerful system but we were getting to the point where it took a full-time engineer to build an observability layer around Temporal. Integrating workflows in an intuitive way with OpenTelemetry and logging was surprisingly non-arbitrary. We wanted to build more of a Vercel-like experience for managing workflows. We have a section on the docs page for durable execution [1], also see the comment on HN [2]. Like I mention in that comment, we still have a long way to go before users can write a full workflow in code in the same style as a Temporal workflow, users either define the execution path ahead of time or invoke a child workflow from an existing workflow. This is also something that requires customization for each SDK - like Temporal's custom asyncio event loop in their Python SDK [3]. We don't want to roll this out until we can be sure about compatibility with the way most people write their functions. [1] https://docs.hatchet.run/home/features/durable-execution https://docs.hatchet.run/home/features/durable-execution [2] https://news.ycombinator.com/item?id=39643881 https://news.ycombinator.com/item?id=39643881 [3] https://github.com/temporalio/sdk-python https://github.com/temporalio/sdk-python
  - bicijay 3y ago
    Well, you just got an user. Love the concept of temporal, but i can't justify the overhead you need with infra to make it work for the upper guys... And the cloud offering is a bit expensive for small companies.
    - mfateev 3y ago
kcorbitt 3y ago
I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple languages and decent built-in observability) for like... 3 years. Every 6 months I'll check in and see if someone has built it yet, evaluate the alternatives, and come away disappointed. One important feature request that probably would block our adoption: one reason why I prefer a postgres-backed queue over eg. Redis is just to simplify our infra by having fewer servers and technologies in the stack. Adding in RabbitMQ is definitely an extra dependency I'd really like to avoid. (Currently we've settled on graphile-worker which is fine for what it does, but leaves a lot of boxes unchecked.)
- abelanger 3y ago
  Thank you, appreciate the kind words! What boxes are you looking to check? Yes, I'm not a fan of the RabbitMQ dependency either - see here for the reasoning: https://news.ycombinator.com/item?id=39643940 https://news.ycombinator.com/item?id=39643940. It would take some work to replace this with listen/notify in Postgres, less work to replace this with an in-memory component, but we can't provide the same guarantees in that case.
  - jaggederest 3y ago
    I come to this only as an interested observer, but my experience with listen/notify is that it outperforms rabbitmq/kafka in small to medium operations and has always pleasantly surprised me. You might find out it's a little easier than you think to slim your dependency stack down.
    - hosh 3y ago
      How do you handle things when no listeners are available to be notified?
      - abelanger 3y ago
        Presumably there'd be a messages table that you listen/notify on, and you'd replay messages that weren't consumed when a listener rejoins. But yeah, this is the overhead I was referencing.
toddmorey 3y ago
I need task queues where the client (web browser) can listen to the progress of the task through completion. I love the simplicity & approachability of Deno queues for example, but I’d need to roll my own way to subscribe to task status from the client. Wondering if perhaps the Postgres underpinnings here would make that possible. EDIT: seems so! https://docs.hatchet.run/home/features/streaming https://docs.hatchet.run/home/features/streaming
- rad_gruchalski 3y ago
  If you need to listen for the progress only, try server-sent events, maybe?: https://en.wikipedia.org/wiki/Server-sent_events https://en.wikipedia.org/wiki/Server-sent_events It's dead simple: an existence of the URI means the topic/channel/whathaveu exists, to access it one needs to know the URI, data streamed but no access to old data, multiple consumers no problem.
- abelanger 3y ago
  Yep, exactly - Gabe has also been thinking about providing per-user signed URLs to task executions so clients can subscribe more easily without a long-lived token. So basically, you would start the workflow from your API, and pass back the signed URL to the client, where we would then provide a React hook to get task updates automatically. We need this ourselves once we open our cloud instance up to self-serve, since we want to provision separate queues per user, with a Hatchet workflow of course.
  - toddmorey 3y ago
    Awesome to hear!
jerrygenser 3y ago
Something I really like about some pub/sub systems is Push subscriptions. For example in GCP pub/sub you can have a "subscriber" that is not pulling events off the queue but instead is an http endpoint where events are pushed to. The nice thing about this is that you can use a runtime like cloud run or lambda and allow that runtime to scale based on http requests and also scale to zero. Setting up autoscaling for workers can be a little bit more finicky, e.g. in kubernetes you might set up KEDA autoscaling based on some queue depth metrics but these might need to be exported from rabbit. I suppose you could have a setup where your daemon worker is making http requests and in that sense "push" to the place where jobs are actually running but this adds another level of complexity. Is there any plan to support a push model where you can push jobs into http and some daemons that are holding the http connections opened?
- abelanger 3y ago
  I like that idea, basically the first HTTP request ensures the worker gets spun up on a lambda, and the task gets picked up on the next poll when the worker is running. We already have the underlying push model for our streaming feature: https://docs.hatchet.run/home/features/streaming https://docs.hatchet.run/home/features/streaming. Can configure this to post to an HTTP endpoint pretty easily. The daemon feels fragile to me, why not just shut down the worker client-side after some period of inactivity?
  - jerrygenser 3y ago
    I think it depends on the http runtime. One of the things with cloud run is that if the server is not handling requests, it doesn't get CPU time. So even if the first request is "wake up", it wouldn't get any CPU to poll outside of the request-response cycle. You can configure cloud run to always allocate CPU but it's a lot more expensive. I don't think it would be a good autoscaling story since autoscaling is based on http requests being processed. (maybe can be done via CPU but that's may not be what you want, it may not even be cpu bound)
- alexbouchard 3y ago
  The push queue model has major benefits has you mentioned. We've built Hookdeck (hookdeck.com) on that premise. I hope we see more projects adopt it.
dalberto 3y ago
I'm curious if this supports coroutines at tasks in Python. It's especially useful for genAI, and legacy queues (namely Celery) are lacking in this regard. It would help to see a mapping of Celery to Hatchet as examples. The current examples require you to understand (and buy into) Hatchet's model, but that's hard to do without understanding how it compares to existing solutions.
SCUSKU 3y ago
Looks pretty great! My biggest issue with Celery has been that the observability is pretty bad. Even if you use Celery Flower, it still just doesn’t give me enough insight when I’m trying to debug some problem in production. I’m all for just using Postgres in service of the grug brain philosophy. Will definitely be looking into this, congrats on the launch!
- abelanger 3y ago
  Appreciate it, thank you! We've spent quite a bit of time in the Celery Flower console. Admittedly it's been a while, I'm not sure if they've added views for chains/groups/etc - it was just a linear task view when I used it. A nice thing in Celery Flower is viewing the `args, kwargs`, whereas Hatchet operates on JSON request/response bodies, so some early users have mentioned that it's hard to get visibility into the exact typing/serialization that's happening. Something for us to work on.
- 9dev 3y ago
  I case you’re stuck with Celery for a while: I was hit with this same problem, and solved it by adding a sidecar HTTP server thread to the Python workers that would expose metrics written by the workers into a multithreaded registry. This has been working amazingly well in production for over two years now, and makes it really straightforward to get custom metrics out of a distributed Celery app.
  - kamikaz1k 3y ago
    Any chance you could share more specifics about your solution?
    - 9dev 3y ago
      Here you go: https://stackoverflow.com/questions/75652326/celery-spawn-sidecar-webserver-process/75799358#75799358 https://stackoverflow.com/questions/75652326/celery-spawn-si... Plus some adjacent discussion on GitHub: https://github.com/prometheus/client_python/issues/902 https://github.com/prometheus/client_python/issues/902 Hope that helps!
acaloiar 3y ago
A related lively dicussion from a few months ago: https://news.ycombinator.com/item?id=37636841 https://news.ycombinator.com/item?id=37636841 Long live Postgres queues.
leetrout 3y ago
Just pointing out even though this is a "Show HN" they are, indeed, backed by YC. Is this going to follow the "open core" pattern or will there be a different path to revenue?
- MuffinFlavored 3y ago
  > path to revenue There have to be at least 10 different ways between different cloud providers to run a distributed task queue. Amazon, Azure, GCP Self-hosting RabbitMQ, etc. I'm curious how they are able to convince investors that there is a sizable portion of market they think doesn't already have this solved (or already has it solved and is willing to migrate)
  - Kinrany 3y ago
    There will be space for improvement until every cloud has a managed offering with exactly the same interface. Like docker, postgres, S3.
  - leetrout 3y ago
    I am curious to see where they differentiate themselves on observability on the longer run. Comparing to rabbitmq it should be easier to see what is in the queue itself without mutating it, for instance.
    - MuffinFlavored 3y ago
      https://www.rabbitmq.com/docs/management https://www.rabbitmq.com/docs/management
      - leetrout 3y ago
        Sure, but to see what is in the queue you have to operate on it, mutating it. With this using postgres we can just look in the table.
  - Aeolun 3y ago
    > I'm curious how they are able to convince investors that there is a sizable portion of market they think doesn't already have this solved Is there any task queue you are completely happy with? I use Redis, but it’s only half of the solution.
Fiahil 3y ago
How does this compare to ZeroMQ (ZMQ) ? https://zeromq.org/ https://zeromq.org/
- vector_spaces 3y ago
  Not the OP or familiar with Hatchet, but generally ZeroMQ is a bit lower down in the stack -- it's something you'd build a distributed task queue or protocol on top of, but not something you'd usually reach for if you needed one for a web service or similar unless you had very special requirements and a specific, careful design in mind. This tool comes with more bells and whistles and presumably will be more constrained in what you can do with it, where ZeroMQ gives you the flexibility to build your own protocol. In principle they have many of the same use cases, like how you can buy ready made whipped cream or whip up your own with some heavy cream and sugar -- one approach is more constrained but works for most situations where you need some whipped cream, and the other is a lot more work and somewhat higher risk (you can over whip your cream and end up with butter), but you can do a lot more with it.
- jeremyjh 3y ago
  ZeroMQ is a library that implements an application layer network protocol. Hatchet is a distributed job server with durability and transaction semantics. Two completely different things at very different levels of the stack. ZeroMQ supports fan-out messaging and other messaging patterns that could maybe be used as part of a job server, but it doesn't have anything to say about durability, retries, or other concerns that job servers take care of, much less a user interface.
Kinrany 3y ago
With NATS in the stack, what's the advantage over using NATS directly?
- abelanger 3y ago
  I'm assuming specifically you mean Nex functions? Otherwise NATS gives you connectivity and a message queue - it doesn't (or didn't) have the concept of task executions or workflows. With regards to Nex -- it isn't fully stable and only supports Javascript/Webassembly. It's also extremely new, so I'd be curious to see how things stabilize in the coming year.
  - rapnie 3y ago
    I recently found Nex in the context of Wasmcloud [0] and ability for it to support long-running tasks/workflows. Impression that indeed Nex needs a good time to mature still. There was also a talk [1] about using Temporal here. For Hatchet it may be interesting to check it out (note: I am not affiliated with Wasmcloud, nor currently using it). [0] https://wasmcloud.com https://wasmcloud.com [1] https://www.temporal.io/replay/videos/zero-downtime-deploys-with-temporal-and-wasmcloud https://www.temporal.io/replay/videos/zero-downtime-deploys-...
  - bruth 3y ago
    (Disclaimer: I am a NATS maintainer and work for Synadia) The parent comment may have been referring to the fact that NATS has support for durable (and replicated) work queue streams, so those could be used directly for queuing tasks and having a set of workers dequeuing concurrently. And this is regardless if you would want to use Nex or not. Nex is indeed fairly new, but the team on is iterating on it quickly and we are dog-fooding it internally to keep stabilizing it. The other benefits of NATS is the built-in multi-tenancy which would allow for distinct applications/teams/contexts to have an isolated set of streams and messaging. It acts as a secure namespace. NATS supports clustering within a region or across regions. For example, Synadia hosts a supercluster in many different regions across the globe and across the three major cloud providers. As it applies to distributed work queues, you can place work queue streams in a cluster within a region/provider closest to the users/apps enqueuing the work, and then deploy workers in the same region for optimizing latency of dequeuing and processing. Could be worth a deeper look on how much you could leverage for this use case.
sroussey 3y ago
Ah nice! I am writing a job queue this weekend for a DAG based task runner, so timing is great. I will have a look. I don't need anything too big, but I have written some stuff for using PostgreSQL (FOR UPDATE SKIP LOCKED for the win), sqlite, and in-memory, depending on what I want to use it for. I want the task graph to run without thinking about retries, timeouts, serialized resources, etc. Interested to look at your particular approach.
zwaps 3y ago
You say this is for generative AI. How do you distribute inference across workers? Can one use just any protocol and how does this work together with the queue and fault tolerance? Could not find any specifics on generative AI in your docs. Thanks
- abelanger 3y ago
  This isn't built specifically for generative AI, but generative AI apps typically have architectural issues that are solved by a good queueing system and worker pool. This is particularly true once you start integrating smaller, self-hosted LLMs or other types of models into your pipeline. > How do you distribute inference across workers? In Hatchet, "run inference" would be a task. By default, tasks get randomly assigned to workers in a FIFO fashion. But we give you a few options for controlling how tasks get ordered and sent. For example, let's say you'd like to limit users to 1 inference task at a time per session. You could do this by setting a concurrency key "<session-id>" and `maxRuns=1` [1]. This means that for each session key, you only run 1 inference task. The purpose of this would be fairness. > Can one use just any protocol We handle the communication between the worker and the queue through a gRPC connection. We assume that you're passing JSON-serializable objects through the queue. [1] https://docs.hatchet.run/home/features/concurrency/round-robin https://docs.hatchet.run/home/features/concurrency/round-rob...
  - zwaps 3y ago
    Got it, so the underlying infrastructure (the inference nodes, if you wish) would be something to be solved outside of Hatched, but it would then allow to schedule inference tasks per user with limits.
hinkley 3y ago
It’s been about a dozen years since I heard someone assert that some CI/CD services were the most reliable task scheduling software for periodic tasks (far better than cron). Shouldn’t the scheduling be factored out as a separate library? I found that shocking at the time, if plausible, and wondered why nobody pulled on that thread. I suppose like me they had bigger fish to fry.
- abelanger 3y ago
  This reminds me of: https://news.ycombinator.com/item?id=28234057 https://news.ycombinator.com/item?id=28234057 If you're saying that the scheduling in Hatchet should be a separate library, we rely on go-cron [1] to run cron schedules. [1] https://github.com/go-co-op/gocron https://github.com/go-co-op/gocron
- lelanthran 3y ago
  Honestly, I'm doing something like that right now, just not in a position to show. All I want is a simple way to specify a tree of jobs to run to do things like checkout a git branch, build it, run the tests, then install the artifacts. Or push a new static website to some site. Or periodically do something. My grug brain simply doesn't want to deal with modern way of doing $SHIT. I don't need to manage a million different tasks per hour, so scaling vertically is acceptable to me, and the benefits of scaling horizontally simply don't appear in my use cases.
cybice 3y ago
Why Hatchet might be better than Windmill: Windmill uses the same approach in PostgreSQL, very fast and has an incredibly good UI.
treesciencebot 3y ago
Latency is really important and that is honestly why we re-wrote most of this stuck ourselves but the project with the gurantee of 25ms< looks interesting. I wish there was an "instant" mode where enough workers are available it could just do direct placement.
- abelanger 3y ago
  To be clear, the 25ms isn't a guarantee. We have a load testing CLI [1] and the secondary steps on multi-step workflows are in the range of 25ms, while the first steps are in the range of 50ms, so that's what I'm referencing. There's still a lot of work to do for optimization though, particularly to improve the polling interval if there aren't workers available to run the task. Some people might expect to set a max concurrency limit of 1 on each worker and have each subsequent workflow take 50ms to start, which isn't be the case at the moment. [1] https://github.com/hatchet-dev/hatchet/tree/main/examples/loadtest/cli https://github.com/hatchet-dev/hatchet/tree/main/examples/lo...
moribvndvs 3y ago
One repeat issue I’ve had with my past position is need to schedule an unlimited number of jobs, often months to year from now. Example use case: a patient schedules an appointment for a follow up in 6 months, so I schedule a series of appointment reminders in the days leading up to it. I might have millions of these jobs. I started out by just entering a record into a database queue and just polling every few seconds. Functional, but our IO costs for polling weren’t ideal, and we wanted to distribute this without using stuff like schedlock. I switched to Redis but it got complicated dealing with multiple dispatchers, OOM issues, and having to run a secondary job to move individual tasks in and out of the immediate queue, etc. I had started looking at switching to backing it with PG and SKIP LOCKED, etc. but I’ve changed positions. I can see a similar use case on my horizon wondered if Hatchet would be suitable for it.
- herval 3y ago
  why do you need to schedule things 6 months in advance, instead of, say, check everything that needs notifications in a rolling window (eg 24h ahead) and schedule those?
  - moribvndvs 3y ago
    Well, it was a dumbed down example. In that particular case, appointments can be added, removed, or moved at any moment, so I can’t just run one job every 24 hours to tee up the next day’s work and leave it at that. Simply polling the database for messages that are due to go out gives me my just-in-time queue, but then I need to build out the work to distribute it, and we didn’t like the IO costs. I did end up moving it Redis and basically ZADD an execution timestamp and job ID, then ZRANGEBYSCORE at my desired interval and remove those jobs as I successfully distribute them out to workers. I then set a fence time. At that time a job runs to move stuff that should have ran but didn’t (rare, thankfully) into a remediation queue, and load the next block of items that should run between now + fence. At the service level, any items with a scheduled date within the fence gets ZADDed after being inserted into the normal database. Anything outside the fence will be picked up at the appropriate time. This worked. I was able to ramp up the polling time to get near-real time dispatch while also noticeably reducing costs. Problems were some occasional Redis issues (OOM and having to either a keep bumping up the Redis instance size or reduce the fence duration), allowing multiple pollers for redundancy and scale (I used schelock for that :/), and occasionally a bug where the poller craps out in the middle of the Redis work resulting in at least once SLA which required downstream protections to make sure I don’t send the same message multiple time to the patient. Again, it all works but I’m interested in seeing if there are solutions that I don’t have to hand roll.
ctoth 3y ago
My only question is why did you call it Hatchet if it doesn't cut down on your logs? I'll show myself out.
- abelanger 3y ago
  Really have an axe to grind with this comment...
fuddle 3y ago
Looks great! Do you publish pricing for your cloud offering? For the self hosted option, are there plans to create a Kubernetes operator? With an MIT license do you fear Amazon could create a Amazon Hatchet Service sometime in the future?
- abelanger 3y ago
  Thank you! > Do you publish pricing for your cloud offering? Not yet, we're rolling out the cloud offering slowly to make sure we don't experience any widespread outages. As soon as we're open for self-serve on the cloud side, we'll publish our pricing model. > For the self hosted option, are there plans to create a Kubernetes operator? Not at the moment, our initial plan was to help folks with a KEDA autoscaling setup based on Hatchet queue metrics, which is something I've done with Sidekiq queue depth. We'll probably wait to build a k8s operator after our existing Helm chart is relatively stable. > With an MIT license do you fear Amazon could create a Amazon Hatchet Service sometime in the future? Yes. The question is whether that risk is worth the tradeoff of not being MIT-licensed. There are also paths to getting integrated into AWS marketplace we'll explore longer-term. I added some thoughts here: https://news.ycombinator.com/item?id=39646788 https://news.ycombinator.com/item?id=39646788.
rheckart 3y ago
Any plans for SDKs outside the current three? .NET Core & Java would be interesting to see..
- abelanger 3y ago
  Not at the moment - the biggest ask has been Rails, but on the other hand Sidekiq is so beloved that I'm not sure it makes sense at the moment. We have our hands very full with the 3 SDKs, though I'd love for us to support a community-backed SDK. If anyone's interested in working on that, feel free to message us in the Discord.
mfrye0 3y ago
I've been looking for this exact thing for awhile now. I'm just starting to dig into the docs and examples, and I have a question on workflows. I have an existing pipeline that runs tasks across two K8 clusters and share a DB. Is it possible to define steps in a workflow where the step run logic is setup to run elsewhere? Essentially not having an inline run function defined, and another worker process listening for that step name.
- abelanger 3y ago
  This depends on the SDK - both Typescript and Golang support a `registerAction` method on the worker which basically let you register a single step to only run on that worker. You would then call `putWorkflow` programmatically before starting the worker. Steps are distributed by default so they run on the workers which have registered them. Happy to provide a more concrete example for the language you're using.
  - mfrye0 3y ago
    Perfect. Yeah, we're using both, but mainly TS. We'll test that out.
fcsp 3y ago
> Hatchet is built on a low-latency queue (25ms average start) That seems pretty long - am I misunderstanding something? By my understanding this means the time from enqueue to job processing, maybe someone can enlighten me.
- mhh__ 3y ago
  It's only a few billion instructions on a decent sized server these days
  - spenczar5 3y ago
    Damn, I want one of these 100GHz CPUs you have, that sounds great. I think you mean million :)
    - jlokier 3y ago
      You'd be surprised. 1 billion instructions in 25ms is realistic these days. My laptop can execute about 400 billion CPU instructions per second on battery. That's about 10 billion instructions in 25ms. Ihat's the CPU alone, i.e. not including the GPU which would increase the total considerably. Also not counting SIMD lanes as separate: The count is bona fide assembly language instructions. It comes from cores running at ~4GHz, 8 issued instructions per clock, times 12 cores, plus 4 additional "efficiency" cores adding a bit more. People have confirmed by measurement the 8 instructions per clock is achievable (or close) in well-optimised code. Average code is more like 2-3 per cycle. Only for short periods as the CPU is likely to get hot and thermally throttle even with its fan. But when it throttles it'll still exceed 1 billion in 25ms. For perspective on how far silicon has come, the GPU on my laptop is reported to do about 14 trillion floating-point 32-bit calculations per second.
    - mhh__ 3y ago
      My ipad has 8 cores executing about 4 to 6 billion instructions a second these days (3GHz at a most ipc of about two)
- abelanger 3y ago
beerkat 3y ago
How does this compare to River Queue (https://riverqueue.com/ https://riverqueue.com/)? Besides the additional Python and TS client libraries.
- abelanger 3y ago
  The underlying queue is very similar. See this comment, which details how we're different from a library client: https://news.ycombinator.com/item?id=39644327 https://news.ycombinator.com/item?id=39644327. We also have the concept of workflows, which last I checked doesn't exist in River. I'm personally very excited about River and I think it fills an important gap in the Go ecosystem! Also now that sqlc w/ pgx seems to be getting more popular, it's very easy to integrate.
krawczstef 3y ago
Can you explain why you chose every function to take in context? https://github.com/hatchet-dev/hatchet/blob/main/python-sdk/examples/dag/worker.py https://github.com/hatchet-dev/hatchet/blob/main/python-sdk/... This seems like a lot of boiler plate to write functions with to me (context I created http://github.com/DAGWorks-Inc/hamilton http://github.com/DAGWorks-Inc/hamilton).
- abelanger 3y ago
  We did it because there are methods that should be accessed which don't map to `args` cleanly. For example, we let users call `context.log`, `context.done` (to determine whether to return on cancellation) or `context.step_output` (to dynamically access a parent's step output). Perhaps there's a more pythonic way to do this? Admittedly this is a pattern we adapted from Go.
  - kamikaz1k 3y ago
    Probably just have it attached to self, like self.context But nbd IMHO
    - krawczstef 3y ago
      yep nbd, but :/
  - krawczstef 3y ago
    you could just make them optional arguments that you inject if they're declared. Happy to chat more. With Hamilton we could actually build an alternative way to describe your API pretty easily...
iangregson 3y ago
I love this idea. I wish it existed a few years ago when I did a not so good job of implementing a distributed DAG processing system :D Looking forward to trying it out!
Nukesor 3y ago
Hey @abelanger, I got a few feature request for Pueue that were out of the scope as they didn't fit Pueue's vision, but seem to fit hatchet quite well (e.g. complex scheduling functionality and multi-agent support) :) One thing I'm missing from your website however, is an actual view from how the interface looks like, what does the actual user interface look like. Having the possibility to schedule stuff in a smart way is nice and all, but how do you *overlook* it? It's important to get a good overview of how your tasks perform. Once I'm convinced that this is actually a useful piece of software, I would like to reference you in the Readme of Pueue as a alternative for users that need more powerful scheduling features (or multi-client support) :) Would that be ok for you?
- abelanger 3y ago
  Pueue looks cool, it's not an alternative to Hatchet though - looks like it's meant to be run in the terminal or by a user? We're very much meant to run in an application runtime. Like I mentioned here [1], we'll expand our comparison section over time. If Pueue's an alternative people are asking about, we'll definitely put it in there. > Having the possibility to schedule stuff in a smart way is nice and all, but how do you overlook it? It's important to get a good overview of how your tasks perform. I'm not sure what you mean by this. Perhaps you're referring to this - https://news.ycombinator.com/item?id=39647154 https://news.ycombinator.com/item?id=39647154 - in which case I'd say: most software is far from perfect. Our scheduling works but has limitations and is being refactored before we advertise it and build it into our other SDKs. [1] https://news.ycombinator.com/item?id=39643631 https://news.ycombinator.com/item?id=39643631
peterisdirsa 3y ago
This is not a viable product, it's a feature
hannasm 3y ago
Seems like this summary should be in the README
CoolCold 3y ago
From your experience, what would be a good way for doing Postgres Master-Master ? My understanding that Postgres Professional/EnterpriseDB based solutions provide reliable M-M and those are proprietary.
Yanael 3y ago
Looks very promising. Recently, I built an asynchronous DAG executor in Python, and I always felt I was reinventing the wheel, but when looking for a resilient and distributed DAG executor, nothing was really meeting the requirements. The feature set is appealing. Wondering if adding/removing/skipping nodes to the DAG dynamically at runtime is possible.
- leafmeal 3y ago
  a little late now, but I wonder if https://github.com/DataBiosphere/toil https://github.com/DataBiosphere/toil might meet your requirements
  - Yanael 3y ago
    it's somehitng interesting I will have a closer look thanks
neonsunset 3y ago
[flagged]
serbrech 3y ago
I wish that this was just a sdk built on top of a provider/standard. Amqp 1.0 is a standard protocol. You can build all this without being tied to a product or to rabbitMQ, with a storage provider and a amqp protocol layer.
welder 3y ago
Related, I also wrote my own distributed task queue in Python [0] and TypeScript [1] with a Show HN [2]. Time it took was about a week. I like your features, but it was easy to write my own so I'm curious how you're building a money making business around an open source product. Maybe the fact everyone writes their own means there's no best solution now, so you're trying to be that and do paid closed source features for revenue? [0] https://github.com/wakatime/wakaq https://github.com/wakatime/wakaq [1] https://github.com/wakatime/wakaq-ts https://github.com/wakatime/wakaq-ts [2] https://news.ycombinator.com/item?id=32730038 https://news.ycombinator.com/item?id=32730038
- abelanger 3y ago
  Nice, Waka looks cool! I've talked a bit about the tradeoffs with library-mode pollers, for example here: https://news.ycombinator.com/item?id=39644327 https://news.ycombinator.com/item?id=39644327. Which isn't to say they don't make sense, but scaling wise I think there can be some drawbacks. > I'm curious how you're building a money making business around an open source product. We'd like to make money off of our cloud version. See the comment on pricing here - https://news.ycombinator.com/item?id=39653084 https://news.ycombinator.com/item?id=39653084 - which also links to other comments about pricing, sorry about that.
  - welder 3y ago
    Thanks. There's definitely a need for this, hence why I built WakaQ. Most distributed task queues have bugs or lack features and are overly complex. Would have been nice to find one I could have used instead of building my own. To be transparent, had Hatchet been around I probably would have self-hosted unless your cloud pricing gave similar throughput for the price I get on DigitalOcean. I'm unique, as a bootstrapped solo company. Maybe Hatchet can be the right solution for others. Keep the momentum going!
wodenokoto 3y ago
Since these are task executions in a DAG, to what degree does it compete with dagster or airflow? I get that I can’t define the task with Hatchet, but if I already want to separate my DAG from my tasks, is this a viable option?
- abelanger 3y ago
  It can be used as an alternative to dagster or airflow but doesn't have the prebuilt connectors that airflow offers. And yes, there are ways to reuse tasks across workflows, but the docs for that aren't quite there yet. The key is to call a `registerAction` method and create the workflow programmatically - but we have some work to do before we publicize this pattern (for one, removing the overloading of the term action, function, step and task). We'll be posting updates and announcements in the Discord - and the Github in our releases - I'd expect that we document this pattern pretty soon.
notpushkin 3y ago
Congrats on the launch! You say Celery can use Redis or RabbitMQ as a backend, but I've also used it with Postgres as a broker successfully, although on a smaller scale (just a single DB node). It's undocumented, so definitely won't recommend anybody using this in production now, but seems to still work fine. [1] How does Hatchet compare to this setup? Also, have you considered making a plugin backend for Celery, so that old systems can be ported more easily? [1]: https://stackoverflow.com/a/47604045/1593459 https://stackoverflow.com/a/47604045/1593459
adeptima 3y ago
Exciting time for distributed, transactional task queue projects built on the top of PostgreSQL! Here's the most heavily upvoted in the past 12 months Hatchet https://news.ycombinator.com/item?id=39643136 https://news.ycombinator.com/item?id=39643136 Inngest https://news.ycombinator.com/item?id=36403014 https://news.ycombinator.com/item?id=36403014 Windmill https://news.ycombinator.com/item?id=35920082 https://news.ycombinator.com/item?id=35920082 HN comments on Temporal.io https://github.com/temporalio https://github.com/temporalio https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=Temporal.io&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Internally we rant about the complexity of the above projects vs using transactional job queues libs like: river https://news.ycombinator.com/item?id=38349716 https://news.ycombinator.com/item?id=38349716 neoq: [https://github.com/acaloiaro/neoq](https://github.com/acaloiaro/neoq) https://github.com/acaloiaro/neoq](https://github.com/acaloi... gue: [https://github.com/vgarvardt/gue](https://github.com/vgarvardt/gue) https://github.com/vgarvardt/gue](https://github.com/vgarvar... Deep inside can't wait to see some like ThePrimeTimeagen to review it ;) https://www.youtube.com/@ThePrimeTimeagen https://www.youtube.com/@ThePrimeTimeagen
- deleted 3y ago
  [deleted]
wereHamster 3y ago
Does it (or will it, ie. is it planned) support delayed execution? eg. I have a task that I want to run at a certain time in the future?
- abelanger 3y ago
  It is planned - see here for more details: https://news.ycombinator.com/item?id=39646300 https://news.ycombinator.com/item?id=39646300. We still need to do some work on this feature though, we'll make sure to document it when it's well-supported.
rubenfiszel 3y ago
Ola, fellow YC founders. Surely you have seen Windmill since you refer to it in the comments below. It looks like Hatchet, being a lot more recent, has currently a subset of what Windmill offers, albeit with a focus solely on the task queue and without the self-hosted enterprise focus. So it looks more like a competitor to Inngest than of Windmill. We released workflows as code last week which was the primary differentiator with other workflow engines and us so far: https://www.windmill.dev/docs/core_concepts/workflows_as_code https://www.windmill.dev/docs/core_concepts/workflows_as_cod... The license is more permissive than ours MIT vs AGPLv3, and you're using Go vs Rust for us, but other than that the architecture looks extremely similar, also based mostly on Postgres with the same insights than us: it's sufficient. I'm curious where do you see the main differentiator long-term.
- HoyaSaxa 3y ago
  No connection to either company, but for what it’s worth I’d never in a million years consider Windmill and this product to be direct competitors. We’ve had a lot of pain with celery and Redis over the years and Hatchet seems to be a pretty compelling alternative. I’d want to see the codebase stabilize a bit before seriously considering it though. And frankly I don’t see a viable path to real commercialization for them so I’d only consider it if everything you needed really was MIT licensed. Windmill is super interesting but I view it as the next evolution of something like Zapier. Having a large corpus of templates and integrations is the power of that type of product. I understand that under the hood it is a similar paradigm, but the market positioning is rightfully night and day. And I also do see a path to real commercialization of the Windmill product because of the above.
  - rubenfiszel 3y ago
    Windmill is used by large enterprises to run critical jobs that require a predefined amount of resources and can run for months if needed, stream their logs, written in code at scale with upmost reliability, throughput and lowest overhead. The only insight from Zapier is how easy it is to develop new workflows. I understand our positioning is not clear on our landing (and we are working on it), but my read of hatched is that what they put forward is mostly a durable execution engine for arbitrary code in python/typescript on a fleet of managed workers, which is exactly what Windmill is. We are profitable and probably wouldn't if we were MIT licensed with no enterprise features. From reading their documentation, the implementation is extremely similar, you define workflows as code ahead of time, and then the engine make sure to have them progress reliably on your fleet of workers (one of our customer has 600 workers deployed on edge environments). There are a few minor differences, we implement the workers as generic rust binary that pull the workflows, so you never have to redeploy them to test and deploy new workflows, whereas they have developed SDK for each languages to allow you to define your own deployable workers (which is more similar to Inngest/Temporal). Also we use polling and REST instead of gRPC for communications between workers and servers.
cebert 3y ago
The website for Hatchet and the GitHub repository make it look like a compelling distributed task queue solution. I see from the main website that this appears to have commercial aspirations, but I don’t see any pricing information available. Do you have a pricing model yet? I’d be apprehensive to consider using Hatchet in future projects without knowing how much it costs.
- abelanger 3y ago
  We'd like to make money off Hatchet Cloud, which is in early access - some more on that here [1] and here [2]. Pricing will be transparent once we're open access. Like I mention in that comment, we'd like to keep our repository 100% MIT licensed. I realize this is unpopular among open source startups - and I'm sure there are good reasons for that. We've considered these reasons and still landed on the MIT license. [1] https://news.ycombinator.com/item?id=39647101 https://news.ycombinator.com/item?id=39647101 [2] https://news.ycombinator.com/item?id=39646788 https://news.ycombinator.com/item?id=39646788
jbergstroem 3y ago
Have you considered https://github.com/tembo-io/pgmq https://github.com/tembo-io/pgmq for the queue bit?
pdksam 3y ago
How is this different from cadence by Uber or swf?
stephen 3y ago
Wow, looks great! We currently happily use graphile-worker, and have two questions: > full transactional enqueueing Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs... I get that, but fwiw transactionality of "perform business logic against entities + job enqueue" (both for queuing the job itself, as well as work performed by workers) is the primary reason we're using a PG-based job queue, as then we avoid transactional outboxes for each queue/work step. So, dunno, loosing that would be a big deal/kinda defeat the purpose (for us) of a PG-based queue. 2nd question, not to be a downer, but I'm just genuinely curious as a wanna-be dev infra/tooling engineer, but a) why take funding to build this (it seems bootstrappable? maybe that's naive), and b) why would YC keeping putting money into these "look really neat but ...surely?... will never be the 100x returns/billion dollar companies" dev infra startups? Or maybe I'm over-estimating the size of the return/exit necessary to make it worth their while.
- abelanger 3y ago
  Yeah these are great questions. > Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs... Yeah, it's the latter, though we'd actually like to support the former in the longer term. There's no technical reason we can't write the workflow/task and read from the same table that you're enqueueing with in the same transaction as your application. That's the really exciting thing about the RiverQueue implementation, though it also illustrates how difficult it is to support every PG driver in an elegant way. Transactional enqueueing is important for a whole bunch of other reasons though - like assigning workers, maintaining dependencies between tasks, implementing timeouts. > why take funding to build this (it seems bootstrappable? maybe that's naive) The thesis is that we can help some users offload their tasks infra with a hosted version, and hosted infra is hard to bootstrap. > why would YC keeping putting money into these "look really neat but ...surely?... will never be the 100x returns/billion dollar companies" dev infra startups? I think Cloudflare is an interesting example here. You could probably make similar arguments against a spam protection proxy, which was the initial service. But a lot of the core infrastructure needed for that service branches into a lot of other products, like a CDN or caching layer, or a more compelling, full-featured product like a WAF. I can't speak for YC or the other dev infra startups, but I imagine that's part of the thesis.
iacguy 3y ago
Been following since Hatchet was an OSS TFC alternative. Seems like you guys pivoted. Curious to learn why and how you moved from the earlier value prop to this one?

578 points by abelanger 3y ago

dataangel 3y ago

> Distributed > Built on PostGRES Not what people usually mean by distributed, caveat emptor

abelanger 3y ago
Citus Data would like a word.

topicseed 3y ago

What specific strategies does Hatchet employ to guarantee fault tolerance and enable durable execution? How does it handle partial failures in multi-step workflows?

abelanger 3y ago
Each task in Hatchet is backed by a workflow [1]. Workflows are predefined steps which are persisted in PostgreSQL. If a worker dies or crashes midway through (stops heartbeating to the engine), we reassign tasks (assuming they have retries left). We also track timeouts in the database, which means if we miss a timeout, we simply retry after some amount of time. Like I mentioned in the post, we avoid some classes of faults just by relying on PostgreSQL and persisting each workflow run, so you don't need to time out with distributed locks in Redis, for example, or worry about data loss if Redis OOMs. Our `ticker` service is basically its own worker which is assigned a lease for each step run. We also store the input/output of each workflow step in the database. So resuming a multi-step workflow is pretty simple - we just replay the step with the same input. To zoom out a bit - unlike many alternatives [2], the execution path of a multi-step workflow in Hatchet is declared ahead of time. There are tradeoffs to this approach; it makes it much easier to run a single-step workflow or if you know the workflow execution path ahead of time. You also avoid classes of problems related to workflow versioning, we can gracefully drain older workflow version with a different execution path. It's also more natural to debug and see a DAG execution instead of debugging procedural logic. The clear tradeoff is that you can't try...catch the execution of a single task or concatenate a bunch of futures that you wait for later. Roadmap-wise, we're considering adding procedural execution on top of our workflows concept. Which means providing a nice API for calling `await workflow.run` and capturing errors. These would be a higher-level concept in Hatchet and are not built yet. There are some interesting concepts around using semaphores and durable leases that are relevant here, which we're exploring [3]. [1] https://docs.hatchet.run/home/basics/workflows https://docs.hatchet.run/home/basics/workflows [2] https://temporal.io https://temporal.io [3] https://www.citusdata.com/blog/2016/08/12/state-machines-to-run-databases/ https://www.citusdata.com/blog/2016/08/12/state-machines-to-...
- topicseed 3y ago

nextworddev 3y ago

I’m interested in self hosting this. What’s the recommendation here for state persistence and self healing? Wish there was a guide for a small team who wants to self host before trying managed cloud

abelanger 3y ago
I think we might have had a dead link in the README to our self-hosting guide, here it is: https://docs.hatchet.run/self-hosting https://docs.hatchet.run/self-hosting. The component which needs the highest uptime is our ingestion service [1]. This ingests events from the Hatchet SDKs and is responsible for writing the workflow execution path, and then sends messages downstream to our other engine components. This is a horizontally scalable service and you should run at least 2 replicas across different AZs. Also see how to configure different services for engine components [2]. The other piece of this is PostgreSQL, use your favorite managed provider which has point-in-time restores and backups. This is the core of our self-healing, I'm not sure where it makes sense to route writes if the primary goes down. Let me know what you need for self-hosted docs, happy to write them up for you. [1] https://github.com/hatchet-dev/hatchet/tree/main/internal/services/ingestor https://github.com/hatchet-dev/hatchet/tree/main/internal/se... [2] https://docs.hatchet.run/self-hosting/configuration-options#services-configuration https://docs.hatchet.run/self-hosting/configuration-options#...

sixhobbits 3y ago

One of my favourite spaces and presentation in readme is clear and immediately told me what it is and most of the key information that I usually complain is missing. However I am still missing a section on why this is different than any of the other existing and more mature solutions. What led you to develop this over existing options and what different tradeoffs did you make? Extra points if you can concisely tell me what you do badly that your 'competitors' do well because I don't believe there is a one best solution in this space, it is all tradeoffs

sixhobbits 3y ago
Sorry I am dumb and commented after clicking on the link. I would just add your hn text to the readme as that is exactly what I was looking for
- abelanger 3y ago
  Done [1]. We'll expand this section over time. There are also definite tradeoffs to our architecture - spoke to someone wanting the equivalent 1.5m PutRecord/s in Kinesis, which we're definitely not ready for because we're persist every event + task execution in Postgres. [1] https://github.com/hatchet-dev/hatchet/blob/main/README.md#how-does-this-compare-to-alternatives-celery-bullmq https://github.com/hatchet-dev/hatchet/blob/main/README.md#h...

tzahifadida 3y ago

Why not use postgres listen/notify instead of rabbitmq pub sub.

anentropic 3y ago
It uses Postgres rather than RabbitMQ: https://github.com/hatchet-dev/hatchet?tab=readme-ov-file#how-does-this-compare-to-alternatives-celery-bullmq https://github.com/hatchet-dev/hatchet?tab=readme-ov-file#ho...
- anentropic 3y ago
  I see... apparently it uses both
abelanger 3y ago
When I started on this codebase, we needed to implement some custom exchange logic that maps very neatly to fanout exchanges and non-durable queues in RabbitMQ and weren't built out on our PostgreSQL layer yet. This was a bootstrapping problem. Like I mentioned in the comment, we'd like to switch to pub/sub pattern that lets us distribute our engine over multiple geographies. Listen/notify could be the answer once we migrate to PG 16, though there are some concerns around connection poolers like pg_bouncer having limited support for listen/notify. There's a Github discussion on this if you're curious: https://github.com/hatchet-dev/hatchet/discussions/224 https://github.com/hatchet-dev/hatchet/discussions/224.
- tzahifadida 3y ago
  I use haproxy with go listen notify of one of the libs. It works as long as the connection is up. I.e.i have a timeout of 30 min configured in haproxy. Then you have to assume you lost sync and recheck. That is not that bad every 30min... at least for me. You can configure to never close...

Kluggy 3y ago

In https://docs.hatchet.run/home/quickstart/installation https://docs.hatchet.run/home/quickstart/installation, it says > Welcome to Hatchet! This guide walks you through getting set up on Hatchet Cloud. If you'd like to self-host Hatchet, please see the self-hosted quickstart instead. but the link to "self-hosted quickstart" links back to the same page

abelanger 3y ago
This should be fixed now, here's the direct link: https://docs.hatchet.run/self-hosting https://docs.hatchet.run/self-hosting.

kevinlu1248 3y ago

We're building a webhook services on FastAPI + Celery + Redis + Grafana + Loki and the experience with setting up every service incrementally was miserable, and even then it feels like logs are being dropped and we run into reliability issues. Felt like something like this should exist already but I couldn't find anything at the time. Really excited to see where this takes us!

tasn 3y ago
That's exactly why we built Svix[1]. Building webhooks services, even with amazing tools like FastAPI, Celery and Redis is still a big pain. So we just built a product to solve it. Hatchet looks cool nonetheless. Queues are a pain for many other use-cases too. 1: https://www.svix.com https://www.svix.com

pyrossh 3y ago

How is this different from pg-boss[1]? Other than the distributed part it also seems to use skip locked. [1] https://github.com/timgit/pg-boss https://github.com/timgit/pg-boss

abelanger 3y ago
I haven't used pg-boss, and feature-wise it looks very similar and is an impressive project. The core difference is that pg-boss is a library while Hatchet is a separate service which runs independently of your workers. This service also provides a UI and API for interacting with Hatchet - I don't think pg-boss has those things, so you'd probably have to build out observability yourself. This doesn't make a huge difference when you're at 1 worker, but having each worker poll your database can lead to DB issues if you're not careful - I've seen some pretty low-throughput setups for very long-running jobs using a database with 60 CPUs because of polling workers. Hatchet distributes in two layers - the "engine" and the "worker" layer. Each engine polls the database and fans out to the workers over a long-lived gRPC connection. This reduces pressure on the DB and lets us manage which workers to assign tasks to based on things like max concurrent runs on each worker or worker health.

radus 3y ago

You've explained your value proposition vs. celery, but I'm curious if you also see Hatchet as an alternative to Nextflow/Snakemake which are commonly used in bioinformatics.

bluehadoop 3y ago

How does this compare against Temporal/Cadence/Conductor? Does hatchet also support durable execution? https://temporal.io/ https://temporal.io/ https://cadenceworkflow.io/ https://cadenceworkflow.io/ https://conductor-oss.org/ https://conductor-oss.org/

abelanger 3y ago
It's very similar - I used Temporal at a previous company to run a couple million workflows per month. The gRPC networking with workers is the most similar component, I especially liked that I only had to worry about an http2 connection with mTLS instead of a different broker protocol. Temporal is a powerful system but we were getting to the point where it took a full-time engineer to build an observability layer around Temporal. Integrating workflows in an intuitive way with OpenTelemetry and logging was surprisingly non-arbitrary. We wanted to build more of a Vercel-like experience for managing workflows. We have a section on the docs page for durable execution [1], also see the comment on HN [2]. Like I mention in that comment, we still have a long way to go before users can write a full workflow in code in the same style as a Temporal workflow, users either define the execution path ahead of time or invoke a child workflow from an existing workflow. This is also something that requires customization for each SDK - like Temporal's custom asyncio event loop in their Python SDK [3]. We don't want to roll this out until we can be sure about compatibility with the way most people write their functions. [1] https://docs.hatchet.run/home/features/durable-execution https://docs.hatchet.run/home/features/durable-execution [2] https://news.ycombinator.com/item?id=39643881 https://news.ycombinator.com/item?id=39643881 [3] https://github.com/temporalio/sdk-python https://github.com/temporalio/sdk-python
- bicijay 3y ago
  Well, you just got an user. Love the concept of temporal, but i can't justify the overhead you need with infra to make it work for the upper guys... And the cloud offering is a bit expensive for small companies.
  - mfateev 3y ago

kcorbitt 3y ago

I love your vision and am excited to see the execution! I've been looking for exactly this product (postgres-backed task queue with workers in multiple languages and decent built-in observability) for like... 3 years. Every 6 months I'll check in and see if someone has built it yet, evaluate the alternatives, and come away disappointed. One important feature request that probably would block our adoption: one reason why I prefer a postgres-backed queue over eg. Redis is just to simplify our infra by having fewer servers and technologies in the stack. Adding in RabbitMQ is definitely an extra dependency I'd really like to avoid. (Currently we've settled on graphile-worker which is fine for what it does, but leaves a lot of boxes unchecked.)

abelanger 3y ago
Thank you, appreciate the kind words! What boxes are you looking to check? Yes, I'm not a fan of the RabbitMQ dependency either - see here for the reasoning: https://news.ycombinator.com/item?id=39643940 https://news.ycombinator.com/item?id=39643940. It would take some work to replace this with listen/notify in Postgres, less work to replace this with an in-memory component, but we can't provide the same guarantees in that case.
- jaggederest 3y ago
  I come to this only as an interested observer, but my experience with listen/notify is that it outperforms rabbitmq/kafka in small to medium operations and has always pleasantly surprised me. You might find out it's a little easier than you think to slim your dependency stack down.
  - hosh 3y ago
    How do you handle things when no listeners are available to be notified?
    - abelanger 3y ago
      Presumably there'd be a messages table that you listen/notify on, and you'd replay messages that weren't consumed when a listener rejoins. But yeah, this is the overhead I was referencing.

toddmorey 3y ago

I need task queues where the client (web browser) can listen to the progress of the task through completion. I love the simplicity & approachability of Deno queues for example, but I’d need to roll my own way to subscribe to task status from the client. Wondering if perhaps the Postgres underpinnings here would make that possible. EDIT: seems so! https://docs.hatchet.run/home/features/streaming https://docs.hatchet.run/home/features/streaming

rad_gruchalski 3y ago
If you need to listen for the progress only, try server-sent events, maybe?: https://en.wikipedia.org/wiki/Server-sent_events https://en.wikipedia.org/wiki/Server-sent_events It's dead simple: an existence of the URI means the topic/channel/whathaveu exists, to access it one needs to know the URI, data streamed but no access to old data, multiple consumers no problem.
abelanger 3y ago
Yep, exactly - Gabe has also been thinking about providing per-user signed URLs to task executions so clients can subscribe more easily without a long-lived token. So basically, you would start the workflow from your API, and pass back the signed URL to the client, where we would then provide a React hook to get task updates automatically. We need this ourselves once we open our cloud instance up to self-serve, since we want to provision separate queues per user, with a Hatchet workflow of course.
- toddmorey 3y ago
  Awesome to hear!

jerrygenser 3y ago

Something I really like about some pub/sub systems is Push subscriptions. For example in GCP pub/sub you can have a "subscriber" that is not pulling events off the queue but instead is an http endpoint where events are pushed to. The nice thing about this is that you can use a runtime like cloud run or lambda and allow that runtime to scale based on http requests and also scale to zero. Setting up autoscaling for workers can be a little bit more finicky, e.g. in kubernetes you might set up KEDA autoscaling based on some queue depth metrics but these might need to be exported from rabbit. I suppose you could have a setup where your daemon worker is making http requests and in that sense "push" to the place where jobs are actually running but this adds another level of complexity. Is there any plan to support a push model where you can push jobs into http and some daemons that are holding the http connections opened?

abelanger 3y ago
I like that idea, basically the first HTTP request ensures the worker gets spun up on a lambda, and the task gets picked up on the next poll when the worker is running. We already have the underlying push model for our streaming feature: https://docs.hatchet.run/home/features/streaming https://docs.hatchet.run/home/features/streaming. Can configure this to post to an HTTP endpoint pretty easily. The daemon feels fragile to me, why not just shut down the worker client-side after some period of inactivity?
- jerrygenser 3y ago
  I think it depends on the http runtime. One of the things with cloud run is that if the server is not handling requests, it doesn't get CPU time. So even if the first request is "wake up", it wouldn't get any CPU to poll outside of the request-response cycle. You can configure cloud run to always allocate CPU but it's a lot more expensive. I don't think it would be a good autoscaling story since autoscaling is based on http requests being processed. (maybe can be done via CPU but that's may not be what you want, it may not even be cpu bound)
alexbouchard 3y ago
The push queue model has major benefits has you mentioned. We've built Hookdeck (hookdeck.com) on that premise. I hope we see more projects adopt it.

dalberto 3y ago

I'm curious if this supports coroutines at tasks in Python. It's especially useful for genAI, and legacy queues (namely Celery) are lacking in this regard. It would help to see a mapping of Celery to Hatchet as examples. The current examples require you to understand (and buy into) Hatchet's model, but that's hard to do without understanding how it compares to existing solutions.

SCUSKU 3y ago

Looks pretty great! My biggest issue with Celery has been that the observability is pretty bad. Even if you use Celery Flower, it still just doesn’t give me enough insight when I’m trying to debug some problem in production. I’m all for just using Postgres in service of the grug brain philosophy. Will definitely be looking into this, congrats on the launch!

abelanger 3y ago
Appreciate it, thank you! We've spent quite a bit of time in the Celery Flower console. Admittedly it's been a while, I'm not sure if they've added views for chains/groups/etc - it was just a linear task view when I used it. A nice thing in Celery Flower is viewing the `args, kwargs`, whereas Hatchet operates on JSON request/response bodies, so some early users have mentioned that it's hard to get visibility into the exact typing/serialization that's happening. Something for us to work on.
9dev 3y ago
I case you’re stuck with Celery for a while: I was hit with this same problem, and solved it by adding a sidecar HTTP server thread to the Python workers that would expose metrics written by the workers into a multithreaded registry. This has been working amazingly well in production for over two years now, and makes it really straightforward to get custom metrics out of a distributed Celery app.
- kamikaz1k 3y ago
  Any chance you could share more specifics about your solution?
  - 9dev 3y ago
    Here you go: https://stackoverflow.com/questions/75652326/celery-spawn-sidecar-webserver-process/75799358#75799358 https://stackoverflow.com/questions/75652326/celery-spawn-si... Plus some adjacent discussion on GitHub: https://github.com/prometheus/client_python/issues/902 https://github.com/prometheus/client_python/issues/902 Hope that helps!

acaloiar 3y ago

A related lively dicussion from a few months ago: https://news.ycombinator.com/item?id=37636841 https://news.ycombinator.com/item?id=37636841 Long live Postgres queues.

leetrout 3y ago

Just pointing out even though this is a "Show HN" they are, indeed, backed by YC. Is this going to follow the "open core" pattern or will there be a different path to revenue?

MuffinFlavored 3y ago
> path to revenue There have to be at least 10 different ways between different cloud providers to run a distributed task queue. Amazon, Azure, GCP Self-hosting RabbitMQ, etc. I'm curious how they are able to convince investors that there is a sizable portion of market they think doesn't already have this solved (or already has it solved and is willing to migrate)
- Kinrany 3y ago
  There will be space for improvement until every cloud has a managed offering with exactly the same interface. Like docker, postgres, S3.
- leetrout 3y ago
  I am curious to see where they differentiate themselves on observability on the longer run. Comparing to rabbitmq it should be easier to see what is in the queue itself without mutating it, for instance.
  - MuffinFlavored 3y ago
    https://www.rabbitmq.com/docs/management https://www.rabbitmq.com/docs/management
    - leetrout 3y ago
      Sure, but to see what is in the queue you have to operate on it, mutating it. With this using postgres we can just look in the table.
- Aeolun 3y ago
  > I'm curious how they are able to convince investors that there is a sizable portion of market they think doesn't already have this solved Is there any task queue you are completely happy with? I use Redis, but it’s only half of the solution.

Fiahil 3y ago

How does this compare to ZeroMQ (ZMQ) ? https://zeromq.org/ https://zeromq.org/

vector_spaces 3y ago
Not the OP or familiar with Hatchet, but generally ZeroMQ is a bit lower down in the stack -- it's something you'd build a distributed task queue or protocol on top of, but not something you'd usually reach for if you needed one for a web service or similar unless you had very special requirements and a specific, careful design in mind. This tool comes with more bells and whistles and presumably will be more constrained in what you can do with it, where ZeroMQ gives you the flexibility to build your own protocol. In principle they have many of the same use cases, like how you can buy ready made whipped cream or whip up your own with some heavy cream and sugar -- one approach is more constrained but works for most situations where you need some whipped cream, and the other is a lot more work and somewhat higher risk (you can over whip your cream and end up with butter), but you can do a lot more with it.
jeremyjh 3y ago
ZeroMQ is a library that implements an application layer network protocol. Hatchet is a distributed job server with durability and transaction semantics. Two completely different things at very different levels of the stack. ZeroMQ supports fan-out messaging and other messaging patterns that could maybe be used as part of a job server, but it doesn't have anything to say about durability, retries, or other concerns that job servers take care of, much less a user interface.

Kinrany 3y ago

With NATS in the stack, what's the advantage over using NATS directly?

abelanger 3y ago
I'm assuming specifically you mean Nex functions? Otherwise NATS gives you connectivity and a message queue - it doesn't (or didn't) have the concept of task executions or workflows. With regards to Nex -- it isn't fully stable and only supports Javascript/Webassembly. It's also extremely new, so I'd be curious to see how things stabilize in the coming year.
- rapnie 3y ago
  I recently found Nex in the context of Wasmcloud [0] and ability for it to support long-running tasks/workflows. Impression that indeed Nex needs a good time to mature still. There was also a talk [1] about using Temporal here. For Hatchet it may be interesting to check it out (note: I am not affiliated with Wasmcloud, nor currently using it). [0] https://wasmcloud.com https://wasmcloud.com [1] https://www.temporal.io/replay/videos/zero-downtime-deploys-with-temporal-and-wasmcloud https://www.temporal.io/replay/videos/zero-downtime-deploys-...
- bruth 3y ago
  (Disclaimer: I am a NATS maintainer and work for Synadia) The parent comment may have been referring to the fact that NATS has support for durable (and replicated) work queue streams, so those could be used directly for queuing tasks and having a set of workers dequeuing concurrently. And this is regardless if you would want to use Nex or not. Nex is indeed fairly new, but the team on is iterating on it quickly and we are dog-fooding it internally to keep stabilizing it. The other benefits of NATS is the built-in multi-tenancy which would allow for distinct applications/teams/contexts to have an isolated set of streams and messaging. It acts as a secure namespace. NATS supports clustering within a region or across regions. For example, Synadia hosts a supercluster in many different regions across the globe and across the three major cloud providers. As it applies to distributed work queues, you can place work queue streams in a cluster within a region/provider closest to the users/apps enqueuing the work, and then deploy workers in the same region for optimizing latency of dequeuing and processing. Could be worth a deeper look on how much you could leverage for this use case.

sroussey 3y ago

Ah nice! I am writing a job queue this weekend for a DAG based task runner, so timing is great. I will have a look. I don't need anything too big, but I have written some stuff for using PostgreSQL (FOR UPDATE SKIP LOCKED for the win), sqlite, and in-memory, depending on what I want to use it for. I want the task graph to run without thinking about retries, timeouts, serialized resources, etc. Interested to look at your particular approach.

zwaps 3y ago

You say this is for generative AI. How do you distribute inference across workers? Can one use just any protocol and how does this work together with the queue and fault tolerance? Could not find any specifics on generative AI in your docs. Thanks

abelanger 3y ago
This isn't built specifically for generative AI, but generative AI apps typically have architectural issues that are solved by a good queueing system and worker pool. This is particularly true once you start integrating smaller, self-hosted LLMs or other types of models into your pipeline. > How do you distribute inference across workers? In Hatchet, "run inference" would be a task. By default, tasks get randomly assigned to workers in a FIFO fashion. But we give you a few options for controlling how tasks get ordered and sent. For example, let's say you'd like to limit users to 1 inference task at a time per session. You could do this by setting a concurrency key "<session-id>" and `maxRuns=1` [1]. This means that for each session key, you only run 1 inference task. The purpose of this would be fairness. > Can one use just any protocol We handle the communication between the worker and the queue through a gRPC connection. We assume that you're passing JSON-serializable objects through the queue. [1] https://docs.hatchet.run/home/features/concurrency/round-robin https://docs.hatchet.run/home/features/concurrency/round-rob...
- zwaps 3y ago
  Got it, so the underlying infrastructure (the inference nodes, if you wish) would be something to be solved outside of Hatched, but it would then allow to schedule inference tasks per user with limits.

hinkley 3y ago

It’s been about a dozen years since I heard someone assert that some CI/CD services were the most reliable task scheduling software for periodic tasks (far better than cron). Shouldn’t the scheduling be factored out as a separate library? I found that shocking at the time, if plausible, and wondered why nobody pulled on that thread. I suppose like me they had bigger fish to fry.

abelanger 3y ago
This reminds me of: https://news.ycombinator.com/item?id=28234057 https://news.ycombinator.com/item?id=28234057 If you're saying that the scheduling in Hatchet should be a separate library, we rely on go-cron [1] to run cron schedules. [1] https://github.com/go-co-op/gocron https://github.com/go-co-op/gocron
lelanthran 3y ago
Honestly, I'm doing something like that right now, just not in a position to show. All I want is a simple way to specify a tree of jobs to run to do things like checkout a git branch, build it, run the tests, then install the artifacts. Or push a new static website to some site. Or periodically do something. My grug brain simply doesn't want to deal with modern way of doing $SHIT. I don't need to manage a million different tasks per hour, so scaling vertically is acceptable to me, and the benefits of scaling horizontally simply don't appear in my use cases.

cybice 3y ago

Why Hatchet might be better than Windmill: Windmill uses the same approach in PostgreSQL, very fast and has an incredibly good UI.

treesciencebot 3y ago

Latency is really important and that is honestly why we re-wrote most of this stuck ourselves but the project with the gurantee of 25ms< looks interesting. I wish there was an "instant" mode where enough workers are available it could just do direct placement.

abelanger 3y ago
To be clear, the 25ms isn't a guarantee. We have a load testing CLI [1] and the secondary steps on multi-step workflows are in the range of 25ms, while the first steps are in the range of 50ms, so that's what I'm referencing. There's still a lot of work to do for optimization though, particularly to improve the polling interval if there aren't workers available to run the task. Some people might expect to set a max concurrency limit of 1 on each worker and have each subsequent workflow take 50ms to start, which isn't be the case at the moment. [1] https://github.com/hatchet-dev/hatchet/tree/main/examples/loadtest/cli https://github.com/hatchet-dev/hatchet/tree/main/examples/lo...

moribvndvs 3y ago

One repeat issue I’ve had with my past position is need to schedule an unlimited number of jobs, often months to year from now. Example use case: a patient schedules an appointment for a follow up in 6 months, so I schedule a series of appointment reminders in the days leading up to it. I might have millions of these jobs. I started out by just entering a record into a database queue and just polling every few seconds. Functional, but our IO costs for polling weren’t ideal, and we wanted to distribute this without using stuff like schedlock. I switched to Redis but it got complicated dealing with multiple dispatchers, OOM issues, and having to run a secondary job to move individual tasks in and out of the immediate queue, etc. I had started looking at switching to backing it with PG and SKIP LOCKED, etc. but I’ve changed positions. I can see a similar use case on my horizon wondered if Hatchet would be suitable for it.

herval 3y ago
why do you need to schedule things 6 months in advance, instead of, say, check everything that needs notifications in a rolling window (eg 24h ahead) and schedule those?
- moribvndvs 3y ago
  Well, it was a dumbed down example. In that particular case, appointments can be added, removed, or moved at any moment, so I can’t just run one job every 24 hours to tee up the next day’s work and leave it at that. Simply polling the database for messages that are due to go out gives me my just-in-time queue, but then I need to build out the work to distribute it, and we didn’t like the IO costs. I did end up moving it Redis and basically ZADD an execution timestamp and job ID, then ZRANGEBYSCORE at my desired interval and remove those jobs as I successfully distribute them out to workers. I then set a fence time. At that time a job runs to move stuff that should have ran but didn’t (rare, thankfully) into a remediation queue, and load the next block of items that should run between now + fence. At the service level, any items with a scheduled date within the fence gets ZADDed after being inserted into the normal database. Anything outside the fence will be picked up at the appropriate time. This worked. I was able to ramp up the polling time to get near-real time dispatch while also noticeably reducing costs. Problems were some occasional Redis issues (OOM and having to either a keep bumping up the Redis instance size or reduce the fence duration), allowing multiple pollers for redundancy and scale (I used schelock for that :/), and occasionally a bug where the poller craps out in the middle of the Redis work resulting in at least once SLA which required downstream protections to make sure I don’t send the same message multiple time to the patient. Again, it all works but I’m interested in seeing if there are solutions that I don’t have to hand roll.

ctoth 3y ago

My only question is why did you call it Hatchet if it doesn't cut down on your logs? I'll show myself out.

abelanger 3y ago
Really have an axe to grind with this comment...

fuddle 3y ago

Looks great! Do you publish pricing for your cloud offering? For the self hosted option, are there plans to create a Kubernetes operator? With an MIT license do you fear Amazon could create a Amazon Hatchet Service sometime in the future?

abelanger 3y ago
Thank you! > Do you publish pricing for your cloud offering? Not yet, we're rolling out the cloud offering slowly to make sure we don't experience any widespread outages. As soon as we're open for self-serve on the cloud side, we'll publish our pricing model. > For the self hosted option, are there plans to create a Kubernetes operator? Not at the moment, our initial plan was to help folks with a KEDA autoscaling setup based on Hatchet queue metrics, which is something I've done with Sidekiq queue depth. We'll probably wait to build a k8s operator after our existing Helm chart is relatively stable. > With an MIT license do you fear Amazon could create a Amazon Hatchet Service sometime in the future? Yes. The question is whether that risk is worth the tradeoff of not being MIT-licensed. There are also paths to getting integrated into AWS marketplace we'll explore longer-term. I added some thoughts here: https://news.ycombinator.com/item?id=39646788 https://news.ycombinator.com/item?id=39646788.

rheckart 3y ago

Any plans for SDKs outside the current three? .NET Core & Java would be interesting to see..

abelanger 3y ago
Not at the moment - the biggest ask has been Rails, but on the other hand Sidekiq is so beloved that I'm not sure it makes sense at the moment. We have our hands very full with the 3 SDKs, though I'd love for us to support a community-backed SDK. If anyone's interested in working on that, feel free to message us in the Discord.

mfrye0 3y ago

I've been looking for this exact thing for awhile now. I'm just starting to dig into the docs and examples, and I have a question on workflows. I have an existing pipeline that runs tasks across two K8 clusters and share a DB. Is it possible to define steps in a workflow where the step run logic is setup to run elsewhere? Essentially not having an inline run function defined, and another worker process listening for that step name.

abelanger 3y ago
This depends on the SDK - both Typescript and Golang support a `registerAction` method on the worker which basically let you register a single step to only run on that worker. You would then call `putWorkflow` programmatically before starting the worker. Steps are distributed by default so they run on the workers which have registered them. Happy to provide a more concrete example for the language you're using.
- mfrye0 3y ago
  Perfect. Yeah, we're using both, but mainly TS. We'll test that out.

fcsp 3y ago

> Hatchet is built on a low-latency queue (25ms average start) That seems pretty long - am I misunderstanding something? By my understanding this means the time from enqueue to job processing, maybe someone can enlighten me.

mhh__ 3y ago
It's only a few billion instructions on a decent sized server these days
- spenczar5 3y ago
  Damn, I want one of these 100GHz CPUs you have, that sounds great. I think you mean million :)
  - jlokier 3y ago
    You'd be surprised. 1 billion instructions in 25ms is realistic these days. My laptop can execute about 400 billion CPU instructions per second on battery. That's about 10 billion instructions in 25ms. Ihat's the CPU alone, i.e. not including the GPU which would increase the total considerably. Also not counting SIMD lanes as separate: The count is bona fide assembly language instructions. It comes from cores running at ~4GHz, 8 issued instructions per clock, times 12 cores, plus 4 additional "efficiency" cores adding a bit more. People have confirmed by measurement the 8 instructions per clock is achievable (or close) in well-optimised code. Average code is more like 2-3 per cycle. Only for short periods as the CPU is likely to get hot and thermally throttle even with its fan. But when it throttles it'll still exceed 1 billion in 25ms. For perspective on how far silicon has come, the GPU on my laptop is reported to do about 14 trillion floating-point 32-bit calculations per second.
  - mhh__ 3y ago
    My ipad has 8 cores executing about 4 to 6 billion instructions a second these days (3GHz at a most ipc of about two)
abelanger 3y ago

beerkat 3y ago

How does this compare to River Queue (https://riverqueue.com/ https://riverqueue.com/)? Besides the additional Python and TS client libraries.

abelanger 3y ago
The underlying queue is very similar. See this comment, which details how we're different from a library client: https://news.ycombinator.com/item?id=39644327 https://news.ycombinator.com/item?id=39644327. We also have the concept of workflows, which last I checked doesn't exist in River. I'm personally very excited about River and I think it fills an important gap in the Go ecosystem! Also now that sqlc w/ pgx seems to be getting more popular, it's very easy to integrate.

krawczstef 3y ago

Can you explain why you chose every function to take in context? https://github.com/hatchet-dev/hatchet/blob/main/python-sdk/examples/dag/worker.py https://github.com/hatchet-dev/hatchet/blob/main/python-sdk/... This seems like a lot of boiler plate to write functions with to me (context I created http://github.com/DAGWorks-Inc/hamilton http://github.com/DAGWorks-Inc/hamilton).

abelanger 3y ago
We did it because there are methods that should be accessed which don't map to `args` cleanly. For example, we let users call `context.log`, `context.done` (to determine whether to return on cancellation) or `context.step_output` (to dynamically access a parent's step output). Perhaps there's a more pythonic way to do this? Admittedly this is a pattern we adapted from Go.
- kamikaz1k 3y ago
  Probably just have it attached to self, like self.context But nbd IMHO
  - krawczstef 3y ago
    yep nbd, but :/
- krawczstef 3y ago
  you could just make them optional arguments that you inject if they're declared. Happy to chat more. With Hamilton we could actually build an alternative way to describe your API pretty easily...

iangregson 3y ago

I love this idea. I wish it existed a few years ago when I did a not so good job of implementing a distributed DAG processing system :D Looking forward to trying it out!

Nukesor 3y ago

Hey @abelanger, I got a few feature request for Pueue that were out of the scope as they didn't fit Pueue's vision, but seem to fit hatchet quite well (e.g. complex scheduling functionality and multi-agent support) :) One thing I'm missing from your website however, is an actual view from how the interface looks like, what does the actual user interface look like. Having the possibility to schedule stuff in a smart way is nice and all, but how do you *overlook* it? It's important to get a good overview of how your tasks perform. Once I'm convinced that this is actually a useful piece of software, I would like to reference you in the Readme of Pueue as a alternative for users that need more powerful scheduling features (or multi-client support) :) Would that be ok for you?

abelanger 3y ago
Pueue looks cool, it's not an alternative to Hatchet though - looks like it's meant to be run in the terminal or by a user? We're very much meant to run in an application runtime. Like I mentioned here [1], we'll expand our comparison section over time. If Pueue's an alternative people are asking about, we'll definitely put it in there. > Having the possibility to schedule stuff in a smart way is nice and all, but how do you overlook it? It's important to get a good overview of how your tasks perform. I'm not sure what you mean by this. Perhaps you're referring to this - https://news.ycombinator.com/item?id=39647154 https://news.ycombinator.com/item?id=39647154 - in which case I'd say: most software is far from perfect. Our scheduling works but has limitations and is being refactored before we advertise it and build it into our other SDKs. [1] https://news.ycombinator.com/item?id=39643631 https://news.ycombinator.com/item?id=39643631

peterisdirsa 3y ago

This is not a viable product, it's a feature

hannasm 3y ago

Seems like this summary should be in the README

CoolCold 3y ago

From your experience, what would be a good way for doing Postgres Master-Master ? My understanding that Postgres Professional/EnterpriseDB based solutions provide reliable M-M and those are proprietary.

Yanael 3y ago

Looks very promising. Recently, I built an asynchronous DAG executor in Python, and I always felt I was reinventing the wheel, but when looking for a resilient and distributed DAG executor, nothing was really meeting the requirements. The feature set is appealing. Wondering if adding/removing/skipping nodes to the DAG dynamically at runtime is possible.

leafmeal 3y ago
a little late now, but I wonder if https://github.com/DataBiosphere/toil https://github.com/DataBiosphere/toil might meet your requirements
- Yanael 3y ago
  it's somehitng interesting I will have a closer look thanks

neonsunset 3y ago

[flagged]

serbrech 3y ago

I wish that this was just a sdk built on top of a provider/standard. Amqp 1.0 is a standard protocol. You can build all this without being tied to a product or to rabbitMQ, with a storage provider and a amqp protocol layer.

welder 3y ago

Related, I also wrote my own distributed task queue in Python [0] and TypeScript [1] with a Show HN [2]. Time it took was about a week. I like your features, but it was easy to write my own so I'm curious how you're building a money making business around an open source product. Maybe the fact everyone writes their own means there's no best solution now, so you're trying to be that and do paid closed source features for revenue? [0] https://github.com/wakatime/wakaq https://github.com/wakatime/wakaq [1] https://github.com/wakatime/wakaq-ts https://github.com/wakatime/wakaq-ts [2] https://news.ycombinator.com/item?id=32730038 https://news.ycombinator.com/item?id=32730038

abelanger 3y ago
Nice, Waka looks cool! I've talked a bit about the tradeoffs with library-mode pollers, for example here: https://news.ycombinator.com/item?id=39644327 https://news.ycombinator.com/item?id=39644327. Which isn't to say they don't make sense, but scaling wise I think there can be some drawbacks. > I'm curious how you're building a money making business around an open source product. We'd like to make money off of our cloud version. See the comment on pricing here - https://news.ycombinator.com/item?id=39653084 https://news.ycombinator.com/item?id=39653084 - which also links to other comments about pricing, sorry about that.
- welder 3y ago
  Thanks. There's definitely a need for this, hence why I built WakaQ. Most distributed task queues have bugs or lack features and are overly complex. Would have been nice to find one I could have used instead of building my own. To be transparent, had Hatchet been around I probably would have self-hosted unless your cloud pricing gave similar throughput for the price I get on DigitalOcean. I'm unique, as a bootstrapped solo company. Maybe Hatchet can be the right solution for others. Keep the momentum going!

wodenokoto 3y ago

Since these are task executions in a DAG, to what degree does it compete with dagster or airflow? I get that I can’t define the task with Hatchet, but if I already want to separate my DAG from my tasks, is this a viable option?

abelanger 3y ago
It can be used as an alternative to dagster or airflow but doesn't have the prebuilt connectors that airflow offers. And yes, there are ways to reuse tasks across workflows, but the docs for that aren't quite there yet. The key is to call a `registerAction` method and create the workflow programmatically - but we have some work to do before we publicize this pattern (for one, removing the overloading of the term action, function, step and task). We'll be posting updates and announcements in the Discord - and the Github in our releases - I'd expect that we document this pattern pretty soon.

notpushkin 3y ago

Congrats on the launch! You say Celery can use Redis or RabbitMQ as a backend, but I've also used it with Postgres as a broker successfully, although on a smaller scale (just a single DB node). It's undocumented, so definitely won't recommend anybody using this in production now, but seems to still work fine. [1] How does Hatchet compare to this setup? Also, have you considered making a plugin backend for Celery, so that old systems can be ported more easily? [1]: https://stackoverflow.com/a/47604045/1593459 https://stackoverflow.com/a/47604045/1593459

adeptima 3y ago

Exciting time for distributed, transactional task queue projects built on the top of PostgreSQL! Here's the most heavily upvoted in the past 12 months Hatchet https://news.ycombinator.com/item?id=39643136 https://news.ycombinator.com/item?id=39643136 Inngest https://news.ycombinator.com/item?id=36403014 https://news.ycombinator.com/item?id=36403014 Windmill https://news.ycombinator.com/item?id=35920082 https://news.ycombinator.com/item?id=35920082 HN comments on Temporal.io https://github.com/temporalio https://github.com/temporalio https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=Temporal.io&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... Internally we rant about the complexity of the above projects vs using transactional job queues libs like: river https://news.ycombinator.com/item?id=38349716 https://news.ycombinator.com/item?id=38349716 neoq: [https://github.com/acaloiaro/neoq](https://github.com/acaloiaro/neoq) https://github.com/acaloiaro/neoq](https://github.com/acaloi... gue: [https://github.com/vgarvardt/gue](https://github.com/vgarvardt/gue) https://github.com/vgarvardt/gue](https://github.com/vgarvar... Deep inside can't wait to see some like ThePrimeTimeagen to review it ;) https://www.youtube.com/@ThePrimeTimeagen https://www.youtube.com/@ThePrimeTimeagen

deleted 3y ago
[deleted]

wereHamster 3y ago

Does it (or will it, ie. is it planned) support delayed execution? eg. I have a task that I want to run at a certain time in the future?

abelanger 3y ago
It is planned - see here for more details: https://news.ycombinator.com/item?id=39646300 https://news.ycombinator.com/item?id=39646300. We still need to do some work on this feature though, we'll make sure to document it when it's well-supported.

rubenfiszel 3y ago

Ola, fellow YC founders. Surely you have seen Windmill since you refer to it in the comments below. It looks like Hatchet, being a lot more recent, has currently a subset of what Windmill offers, albeit with a focus solely on the task queue and without the self-hosted enterprise focus. So it looks more like a competitor to Inngest than of Windmill. We released workflows as code last week which was the primary differentiator with other workflow engines and us so far: https://www.windmill.dev/docs/core_concepts/workflows_as_code https://www.windmill.dev/docs/core_concepts/workflows_as_cod... The license is more permissive than ours MIT vs AGPLv3, and you're using Go vs Rust for us, but other than that the architecture looks extremely similar, also based mostly on Postgres with the same insights than us: it's sufficient. I'm curious where do you see the main differentiator long-term.

HoyaSaxa 3y ago
No connection to either company, but for what it’s worth I’d never in a million years consider Windmill and this product to be direct competitors. We’ve had a lot of pain with celery and Redis over the years and Hatchet seems to be a pretty compelling alternative. I’d want to see the codebase stabilize a bit before seriously considering it though. And frankly I don’t see a viable path to real commercialization for them so I’d only consider it if everything you needed really was MIT licensed. Windmill is super interesting but I view it as the next evolution of something like Zapier. Having a large corpus of templates and integrations is the power of that type of product. I understand that under the hood it is a similar paradigm, but the market positioning is rightfully night and day. And I also do see a path to real commercialization of the Windmill product because of the above.
- rubenfiszel 3y ago
  Windmill is used by large enterprises to run critical jobs that require a predefined amount of resources and can run for months if needed, stream their logs, written in code at scale with upmost reliability, throughput and lowest overhead. The only insight from Zapier is how easy it is to develop new workflows. I understand our positioning is not clear on our landing (and we are working on it), but my read of hatched is that what they put forward is mostly a durable execution engine for arbitrary code in python/typescript on a fleet of managed workers, which is exactly what Windmill is. We are profitable and probably wouldn't if we were MIT licensed with no enterprise features. From reading their documentation, the implementation is extremely similar, you define workflows as code ahead of time, and then the engine make sure to have them progress reliably on your fleet of workers (one of our customer has 600 workers deployed on edge environments). There are a few minor differences, we implement the workers as generic rust binary that pull the workflows, so you never have to redeploy them to test and deploy new workflows, whereas they have developed SDK for each languages to allow you to define your own deployable workers (which is more similar to Inngest/Temporal). Also we use polling and REST instead of gRPC for communications between workers and servers.

cebert 3y ago

The website for Hatchet and the GitHub repository make it look like a compelling distributed task queue solution. I see from the main website that this appears to have commercial aspirations, but I don’t see any pricing information available. Do you have a pricing model yet? I’d be apprehensive to consider using Hatchet in future projects without knowing how much it costs.

abelanger 3y ago
We'd like to make money off Hatchet Cloud, which is in early access - some more on that here [1] and here [2]. Pricing will be transparent once we're open access. Like I mention in that comment, we'd like to keep our repository 100% MIT licensed. I realize this is unpopular among open source startups - and I'm sure there are good reasons for that. We've considered these reasons and still landed on the MIT license. [1] https://news.ycombinator.com/item?id=39647101 https://news.ycombinator.com/item?id=39647101 [2] https://news.ycombinator.com/item?id=39646788 https://news.ycombinator.com/item?id=39646788

jbergstroem 3y ago

Have you considered https://github.com/tembo-io/pgmq https://github.com/tembo-io/pgmq for the queue bit?

pdksam 3y ago

How is this different from cadence by Uber or swf?

stephen 3y ago

Wow, looks great! We currently happily use graphile-worker, and have two questions: > full transactional enqueueing Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs... I get that, but fwiw transactionality of "perform business logic against entities + job enqueue" (both for queuing the job itself, as well as work performed by workers) is the primary reason we're using a PG-based job queue, as then we avoid transactional outboxes for each queue/work step. So, dunno, loosing that would be a big deal/kinda defeat the purpose (for us) of a PG-based queue. 2nd question, not to be a downer, but I'm just genuinely curious as a wanna-be dev infra/tooling engineer, but a) why take funding to build this (it seems bootstrappable? maybe that's naive), and b) why would YC keeping putting money into these "look really neat but ...surely?... will never be the 100x returns/billion dollar companies" dev infra startups? Or maybe I'm over-estimating the size of the return/exit necessary to make it worth their while.

abelanger 3y ago
Yeah these are great questions. > Do you mean transactional within the same transaction as the application's own state? My guess is no (from looking at the docs, where enqueuing in the SDK looks a lot like a wire call and not issuing a SQL command over our application's existing connection pool), and that you mean transactionality between steps within the Hatchet jobs... Yeah, it's the latter, though we'd actually like to support the former in the longer term. There's no technical reason we can't write the workflow/task and read from the same table that you're enqueueing with in the same transaction as your application. That's the really exciting thing about the RiverQueue implementation, though it also illustrates how difficult it is to support every PG driver in an elegant way. Transactional enqueueing is important for a whole bunch of other reasons though - like assigning workers, maintaining dependencies between tasks, implementing timeouts. > why take funding to build this (it seems bootstrappable? maybe that's naive) The thesis is that we can help some users offload their tasks infra with a hosted version, and hosted infra is hard to bootstrap. > why would YC keeping putting money into these "look really neat but ...surely?... will never be the 100x returns/billion dollar companies" dev infra startups? I think Cloudflare is an interesting example here. You could probably make similar arguments against a spam protection proxy, which was the initial service. But a lot of the core infrastructure needed for that service branches into a lot of other products, like a CDN or caching layer, or a more compelling, full-featured product like a WAF. I can't speak for YC or the other dev infra startups, but I imagine that's part of the thesis.

iacguy 3y ago

Been following since Hatchet was an OSS TFC alternative. Seems like you guys pivoted. Curious to learn why and how you moved from the earlier value prop to this one?