7 ms·
DAG Workflow Engine
- blobmty 5mo agoDAG Workflow Engine A production-ready DAG (Directed Acyclic Graph) workflow engine driven by a YAML DSL. Validates, executes, and visualizes workflows with support for parallel execution, retries, conditional branching, batch iteration, and pluggable actions.
- lixtra 5mo agoIdeally, you submit your own project as https://news.ycombinator.com/showhn.html https://news.ycombinator.com/showhn.html
- esafak 5mo agoI don't see any references to existing orchestrators, which are way more complete, so I presume you did this as an exercise? Just seeing YAML used for workflows in this age makes me automatically nope out.
- afshinmeh 5mo agoCurious, what format would you prefer to use to represent a workflow instead of YAML?
- esafak 5mo agoType-safe code. Workflows are not configuration! If I wanted YAML hell I could stick to Github Actions. But that's only the start. There are a lot of other things I would expect of a new workflow orchestrator in 2026 so if you are not comparing yourself to the competition you probably don't know what you're getting yourself into.
- afshinmeh 5mo agoYeah, that makes sense. I looked at a few workflow orchestrators and I'm building something that I will release soon, but my thinking is that the "workflow engine" should be an abstraction that takes the input and executes the steps. "What" you use to define that workflow is probably the SDK layer though, but I can certainly see the value in using type safe code to define as opposed to a YAML file. I'm mainly focusing on the portability aspect of it (e.g. use TS/Python/etc. to define the workflow/steps or just simple a simple YAML file).
- esafak 5mo ago[dead]
- verdverm 5mo agoAre you planning to map those varied definitions onto varied orchestrators?
- afshinmeh 5mo agoSort of. My thinking is that the input to define the workflow should be anything you prefer to use (TS, Go, YAML, etc.) and the orchestrator's job is to model that and execute the job, given your deployment model.
- verdverm 5mo agoThere are a number of widely used orchestrators, it would be nice to deploy to one of those vs a new kid on the block
- afshinmeh 5mo agoI'm mainly looking at Rust based projects and haven't been able to find something to use out of the box, without hacky RPC/Shell execs. Curious if you have any suggestions?
- 5mo ago
- taybin 5mo agoWhat makes it production ready? What's the code coverage on your tests? There are only seven commits in this repo as of this comment.
- Hasnep 5mo agoThe LLM generated the words "production ready" so it must be true!
- tibbar 5mo agoI was expecting to see some verbose LLM output, but actually the code has a distinctly hand-crafted feel. Nice to see! I'm not sure if "production ready" is a safe claim 7 commits in to a project ;)
- afshinmeh 5mo agohttps://github.com/vivekg13186/Daisy-DAG/blob/main/backend/src/api/ai.js#L142 https://github.com/vivekg13186/Daisy-DAG/blob/main/backend/s...
- antonvs 5mo agoI've seen LLMs include that exact "production-ready" claim on code they generate. But of course it gets that from its training data.
- _ZeD_ 5mo agohow it compares to airflow?
- colton_padden 5mo agoWas going to ask the same thing. The orchestration space already has some very well established frameworks like Airflow and Dagster. Would be curious to see the pros and cons.
- saltyoldman 5mo agoI think the future of replacements to well established frameworks written in Python/etc.. are zero dependency binaries (from Rust or Go) that require so little configuration and tuning and they "just work". That being said, that's not this project.
- topspin 5mo agoAgreed. Right now, if I needed "workflow" for a greenfield that could tolerate some risk, I'd look at https://www.restate.dev/ https://www.restate.dev/ which matches your model of a self contained binary.
- peterkelly 5mo agoI've always been of the view that for a workflow language, you should use a proper, turing-complete functional language which gives you all the usual flexiblity for transformations on intermediate data, while also supporting things like automatic parallelisation of things like external, compute-intensive tasks. I recommend checking out https://github.com/peterkelly/rex https://github.com/peterkelly/rex and also my PhD thesis on the topic https://www.pmkelly.net/publications/thesis.pdf https://www.pmkelly.net/publications/thesis.pdf. The gap in flexiblity between DAG-only and a full language designed for the task is a significant one.
- antonvs 5mo agoDo you implement a DAG within your system to act as a kind of well-defined backbone for analysis and execution, or do you dispense with (explicit) DAGs entirely?
- mrauha 5mo agoredun is quite interesting in this regard https://insitro.github.io/redun/ https://insitro.github.io/redun/
- ofrzeta 5mo agoI guess that ship has sailed and also it's maybe nitpicking but I find it a bit unfortunate to call a new programming language "Rex" when there's already "Rexx" for several decades.
- smartmic 5mo agoI wonder, isn‘t any Lisp, be it Clojure, Scheme, etc. not exactly suited for such tasks?
- deleted 5mo ago[deleted]
- Moosifer 5mo ago[dead]
- 5mo ago
- purpleidea 5mo agoHere's a different kind of workflow engine with a proper DSL. It turns out config management is the same problem as workflow engines, if you use my modern definition of config management. https://github.com/purpleidea/mgmt/ https://github.com/purpleidea/mgmt/
- tedchs 5mo agoHow does this compare to Temporal? That seems to be the current baseline for application-oriented workflow engines.
- halfcat 5mo agoTemporal and DBOS are more around the durability guarantees. If you have tasks that are expensive to restart from scratch, or if you have human-in-the-loop approvals, or you have months between steps (e.g. 90-day warranty inspection after installation), you want that durability.
- SkyPuncher 5mo agoI'm working on something similar as a side project. I'm finding frustration with a lack of repeatability in my LLM flows. 90% of my code is AI written, but most of my guidance to LLMs is not particularly specific. It's "make sure you've read this file", "how does that match against existing patterns", "what's the performance like". I've ended up building my workflow engine directly in Python, despite YAML being the default choice for LLMs. I found that YAML had some drawbacks: * LLMs don't have an inherent understanding of YAML conventions. They tend to be overly verbose. Python code solved this because "good" code is generally as short as you need. * YAML isn't really composable. Yes, you can technically compose it, but you'll be fighting the LLM the entire time. Python solved this because the LLM knows how to decouple code. * I want _some_ things to be programatic still. Having Python solves that * Pretty much any programming language would do. Python just feels like the default for LLM-centric code.
- barelysapient 5mo agoMy version of a similar tool; but written in Go with a compile time guarantee. https://github.com/swetjen/daggo https://github.com/swetjen/daggo
- subhobroto 5mo agoThis is a good exercise but IMHO, when you really start using a workflow for production usecases, you need a a proper, turing-complete programming language as a DSL. There used to be a project called Benthos (since acquired and rebranded by Redpanda in 2024) that was amazing, that you might want to gain some inspiration from. However, durable workflows have also gained popular acceptance as functional design reaches a wider audience. While Temporal is the most popular choice when it comes to durable workflows, DBOS (cofounded by the father of PostgreSQL) is my personal favorite. At the moment, orchestration in DBOS has certain gaps - you might very well consider spending your effort on closing those gaps. The value there would be phenomenal!
- FelipeCortez 5mo agoI love Temporal and am DBOS-curious. what do you think DBOS does better?
- subhobroto 5mo agoHi Felipe! Just point your agent at https://docs.dbos.dev/python/prompting https://docs.dbos.dev/python/prompting and give it a go - you can really play around with it as much as you want and solve real problems you care about than me lecturing you about it :) That said, DBOS really makes durable workflows accessible and approachable. Having already used Temporal, I think you're really appreciate how quickly you can get started with DBOS. I forget if they support SQLite but if you have a PostgreSQL server set up, you really don't need anything else to write your first few DBOS durable workflows (vs. needing a Temporal server or cluster) Let me know if I got you interested to try it out. I first learned about Temporal from Mitchell Hashimoto as they were using it for Hashicorp Cloud. Eventually I discovered DBOS and now all my personal projects are on DBOS.
- halfcat 5mo agoThe entire state is (mainly) two tables in Postgres. Maybe 10 tables total if you’re using all the features. There’s something about seeing the ground truth, in full, in one place, when you’re trying to understand it, or troubleshoot it.
- b4rtaz__ 5mo agoIt’s interesting to see something new in this space, especially since some people claim that flowcharts will be replaced by AI automation or AI-generated code. P.S. I'm the author of a similar solution: * https://github.com/nocode-js/sequential-workflow-designer https://github.com/nocode-js/sequential-workflow-designer * https://github.com/nocode-js/sequential-workflow-machine https://github.com/nocode-js/sequential-workflow-machine
- kovariance 5mo agoYAML as a programming language is something I consider as an anti-pattern (see AWS Step Functions). Very difficult to read/debug/test. It's better to use a real programming language that compiles into a DAG (e.g. Temporal, Dagger.io).
- halfcat 5mo ago> better to use a real programming language that compiles into a DAG Fun fact: a DAG, after topological sorting, is a list Many people need the efficiency of running things in parallel. But if you don’t (like if you’re running reporting/ETL stuff overnight), you can skip a lot of the complexity and just run a list of tasks in the right order. Or put another way, before you adopt a DAG orchestrator (and all the time evaluating your options), remember you can just run the same steps as a list and get something shipped, and the DAG stuff is an optimization you can tackle in phase 2.
- zackham 5mo agoI have a project in this space that I've run many thousands of jobs through. It's solid and full featured. Feel free to connect: https://stepwise.run/ https://stepwise.run/
- kamikazechaser 5mo agoJust looking at the features, this is pretty cool!
- panda888888 5mo agoHow is this different from Airflow or commercial data orchestration tools, like Astronomer, Dagster, Prefect, etc.?
- hyperbovine 5mo agoIt's buggier and less functional.
- Keyframe 5mo agoquite a statement considering which were mentioned!
- SOLAR_FIELDS 5mo agoIt's actually a slightly oblong wheel vs a round one
- zaptheimpaler 5mo agoI have several sources of data I want to fetch, retry, process periodically. Like exporting Claude chats into .md files that go to Obsidian, fetching Garmin data from the API and processing it for a custom tool, exporting replays for a game, maybe even running some browser automation to get bank CSVs. I have some ad-hoc python scripts for all of this but no central way to manage them, schedule, handle errors and retries, store the original data and processed versions, resume from the last point etc.. is a workflow engine useful for something like that?
- momojo 5mo agoCheck out Airflow and Dagster. I've used Dagster but I can't compare to airflow. But in terms of DX, I've found Dagster pretty easy to use. Instead of writing their own DSL, they have a python library that lets yo tag your pre-made methods as @ops and and string them together into a DAG.
- halfcat 5mo agoAgree with other response, look at Dagster for this. If you want to roll your own, you build a dependency graph (a dict) of the functions you want to call, Python already has graphlib.TopologicalSorter built in that can do this for you. Throw in logging and the tenacity library for retries and you’re set.
- 535188B17C93743 5mo ago[dead]
- bognition 5mo agoProduction Ready? That is a is a pretty bold claim for a repo that existed for a few days, has 0 issues, PRs, etc...
- halfcat 5mo ago0 open issues. That means it’s production ready, right? Right??
- tbrownaw 5mo agoThese are always a fun couple-day project. :)
- philipodonnell 5mo agoThis particular example aside, I don’t think it being derivative and simplified is necessarily bad. Libraries that are popular today were written for humans and reinforced by LLMs via training. It’s unlikely they represent the ideal interaction surface for an agent. There was a study recently that LLms prefer resumes written by LLMs rather than by humans. Stands to reason they would prefer apis written by LLMs. This is probably the early days of such intentionally simplified agentic semantic primitives like “DAG Workflow” where the answer for why not Temporal is that LLMs prefer different things than humans.
- snthpy 5mo agoYAML no thanks. I want something that uses BPML for actual business workflows.
- yohamta 5mo agoThis is a very interesting project, especially since I've been building a similar declarative workflow engine for over 5 years. With a well-designed YAML schema, it's now possible to build workflows with AI agents. I call this "Vibe workflows." There's no need for humans to write DAGs anymore, yet they remain human-readable. I truly believe this is the future of workflow orchestration. https://github.com/dagucloud/dagu https://github.com/dagucloud/dagu