6 ms·
Show HN: Workflow orchestrator in Golang
A brief overview:
1. Workflows steps share a running context, with access to data they need require.
2. Steps in the workflow (builders) are chained together based on a topologically sorted built from the predefined input & output.
3. No servers spin up (like Conductor/Cadence) - the orchestrator is low level and meant for simplifying business logic.
4. Before/After listeners for each step.
Would love to hear your thoughts and feedback!
- peter_l_downs 3y agoLooks like a great side project, congrats on launching. My basic feedback is that you should set expectations on the project page by answering the following questions: - Why would someone use this instead of Airflow/Cadence/Temporal/Databuilderframework? - What does this look like when it's used? Most frameworks provide some kind of example project, you should too. - Related, but more specifically, what does the `IDataStore` interface contract mean? Beyond the two functions that I have to implement, are there any considerations related to the overall performance/scalability/durability of the system? Would it make sense to use a disk-backed store, or Redis, or Postgres? - How do I observe the system? Which workflows are running, which have failed, what the current state is, etc. Are there metrics? Logs? All of this is based on the assumption you want people to adopt this framework. If it's just a cool side project, that's fine too, but you should probably try to set that expectation in the README.
- purpleidea 3y agoI don't generally believe in orchestrators (they miss the point, things are not single computers and neither is the world) and so I have that feedback here but also for: > Airflow/Cadence/Temporal/Databuilderframework? Which don't really think about modelling non-centralized things. This of course doesn't mean they're not useful, it's just that they don't have what I believe is a good long-term value proposition. I'm incredibly biased because I'm working on programmatic, real-time modelling of distributed systems with https://github.com/purpleidea/mgmt/ https://github.com/purpleidea/mgmt/
- Thaxll 3y agoI don't think you understand what Temporal does.
- harshadmanglani 3y agoCan you elaborate on what you mean by modelling non-centralized things?
- harshadmanglani 3y agoSuper helpful feedback @peter. Thanks a ton! I've noted all of these and I'll modify the README to include them. Thank you for taking the time to go through in such detail :) Given that the current state of a workflow: - is inherently invisible - all we can really check in the DB is if, for a workflow, the available data contains the target data; how would I address observability concerns? This is a function of a lack of workflow states due to the lower levels of abstraction it operates on. User defined workflow states would do the trick, but I suppose that would take writing some more code after integrating the framework.
- peter_l_downs 3y agoI would recommend either updating your framework to allow for instrumentation (logs/metrics/etc) or showing how to add that instrumentation in "user defined workflow states" via an example application. The current pitch, which is "this is a workflow orchestration framework that does not allow for any monitoring or observability" is a complete non-starter. You may want to consider how other projects keep track of workflow state and allow for it to be instrumented.
- harshadmanglani 3y agoFair enough, will do. Thanks!
- scosman 3y agoLove this space. I’ve build a few versions of this in go, glad to see some go native frameworks. Would also love to see some examples. Concerns I’ve had to deal with in the past (if these are solved, examples would go a long way): - max queue size between elements is important. Don’t want the database reader role to outrun the workers, but also never want them desaturated - some dynamic limits for local to a role (limit DB readers to limit connection pool) and global (how much cpu and memory share does a job need before starting)
- harshadmanglani 3y agoThanks for the inputs! I'll add some examples in the documentation. - with control on the database reader yourself, i think you should be able to find a way around saturation/desaturation? - again, the fairly certain you can limit DB readers in the DataStore interface you pass to the orchestrator. I'll think about the cpu and memory share and if there's a way to expose that. A major concern I've always had with workflow orchestrators is the versioning of workflows. Think about long running workflows (>2 days). If you change your workflow logic, what happens to the existing ones that haven't completed yet? Everyone handles this differently, and I've been thinking about a generic way to do this. Thoughts?
- mdaniel 3y agoIt seems you overlooked adding a license so that people can know under what circumstances they can use your project's code
- deleted 3y ago[deleted]
- withinboredom 3y agoIf there isn’t a license, you can’t use it. I think it’s pretty clear.
- brauhaus 3y agoIt's the opposite. Without a license, the default copyright laws apply, meaning that the author retains all rights to their source code and no one may reproduce, distribute, or create derivative works from their work. Source: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/licensing-a-repository https://docs.github.com/en/repositories/managing-your-reposi...
- harshadmanglani 3y agothanks for sharing this. that's what @withinboredom mentioned too :)
- withinboredom 3y agoThat's uhh... exactly what I said?
- harshadmanglani 3y agoLet me add a license, I missed it. Thanks for pointing it out!
- reactordev 3y ago[flagged]
- deleted 3y ago[deleted]
- ksajadi 3y agoThis looks very cool for embedding workflows in a native Go project. There is also Trackman (https://github.com/cloud66-oss/trackman https://github.com/cloud66-oss/trackman) which is mostly built for workflow based commanline execution.
- emmanueloga_ 3y agoSo many tools in this space! This one looks a little bit like go-task, but it seems maybe better for production workflows because if timeout support, while go-task seems more aimed to command line work/makefile replacement. —- https://github.com/go-task/task https://github.com/go-task/task
- harshadmanglani 3y agoThanks for sharing this! There is no support for timeouts yet, but might be a good addition.
- harshadmanglani 3y agoThanks for sharing! I'll check it out.
- liftm 3y agoArgh, using this would cause name collisions in my brain. https://github.com/agersant/polaris/ https://github.com/agersant/polaris/ ^^
- deepakarora3 3y agoNice. It is great to see native lightweight opensource (I hope it is considering that someone said that there is no license file yet) solutions hit this space. For what it's worth, I have built something similar to this but for Java programming language. You can find it here -> https://github.com/americanexpress/unify-flowret https://github.com/americanexpress/unify-flowret. My reason for building something like this was that the product market is just too unwieldy to work with and has multiple layers of complexity which most of the time can be done away with. Just my opinion. On a side note, you will at some point in time have to deal with multi version workflows. I know that this is one feature that limits wide adoption of an orchestrator.
- harshadmanglani 3y agoYes, I've added the license. Thanks for sharing your work Deepak, you have some pretty extensive documentation! Funnily enough, the Golang framework is almost a clone of https://github.com/flipkart-incubator/databuilderframework https://github.com/flipkart-incubator/databuilderframework (another orchestration engine in Java). This is another HN post that extensively covers almost every major orchestrator in the market: https://news.ycombinator.com/item?id=24216317 https://news.ycombinator.com/item?id=24216317 As for multi version workflows, I suppose that will have to be a tradeoff between maintaining somewhat redundant code or adding workflows as WorkflowV1 and WorkflowV2 and stitching together relevant steps in respective versions (reduces redundancy to some extent but won't eliminate)
- harshadmanglani 3y agoAdded docs and an example with docusaurus: https://harshadmanglani.github.io/polaris https://harshadmanglani.github.io/polaris