5 ms·

Aside from this platform, which I've never heard about until now, I'm wondering what others are using in the workflow orchestration space? I'd assume Airflow i

Aside from this platform, which I've never heard about until now, I'm wondering what others are using in the workflow orchestration space? I'd assume Airflow is the most prevalent, but there's also Argo getting quite a bit of momentum lately.

yamrzou 4y ago
In my previous job, I used Dagster. It has served us well. See my comment about it here: https://news.ycombinator.com/item?id=28803117 https://news.ycombinator.com/item?id=28803117
kodablah 4y ago
https://temporal.io/ https://temporal.io/ (disclaimer: I work for Temporal on the Go SDK and upcoming Python SDK)
- wharfjumper 4y ago
  Oooh I hadn't realised there was a Python SDK under development - will take a look.
ricklamers 4y ago
We’re building Orchest (https://github.com/orchest/orchest https://github.com/orchest/orchest) tldr; a GUI for Argo
adamzegelin 4y ago
Cadence: https://cadenceworkflow.io https://cadenceworkflow.io
kaelin 4y ago
abrazensunset 4y ago
"I want to write my orchestration in Python and I'm comfortable hosting my own compute" -> Prefect (lightweight) or Dagster (heavier but featureful) "My team already knows Airflow and/or I want to pay Astronomer a lot of money" -> Airflow "I love YAML and everything is on k8s anyway" -> Argo "I just want something that works out of the box and don't want to host my own compute" -> Shipyard, maybe Orchest "I want a more flexible, generic workflow engine and don't care about writing orchestration in Python" -> Temporal/Cadence "I am very nostalgic" -> Azkaban, Oozie, Luigi "I love clunky Java solutions to data problems" -> Nifi et al "I like to pay for half-managed solutions and late upgrades to a first-generation technology" -> AWS/GCP hosted Airflow options "I am on AWS and it doesn't need to be complicated" -> AWS Step Functions
- herodoturtle 4y ago
  This was a really useful comment. Thank you. I know there's a degree of oversimplification going on here, but there's something to be said for having a simple bullet-list breakdown of all the use-cases - alongside the best tool for each use-case. It is servers as a practical starting point in terms of narrowing down the list of tools (of which there are so many), before one proceeds with a deeper dive into the best fitting tool. Would be great if there were a site that did this sort of thing for all the common architectural needs.
- corrius 4y ago
  +1 to AWS Step Functions, in my last three companies I have built fairly complicated workflows with them and once you get used to them they are very powerful, reliable and cheap. I just wish a little bit more monitoring on top of them but nothing you can not build by yourself.
- otabdeveloper4 4y ago
  I ended up choosing NodeRed at current job. What does that make me? :)

yamrzou 4y ago

In my previous job, I used Dagster. It has served us well. See my comment about it here: https://news.ycombinator.com/item?id=28803117 https://news.ycombinator.com/item?id=28803117

kodablah 4y ago

https://temporal.io/ https://temporal.io/ (disclaimer: I work for Temporal on the Go SDK and upcoming Python SDK)

wharfjumper 4y ago
Oooh I hadn't realised there was a Python SDK under development - will take a look.

ricklamers 4y ago

We’re building Orchest (https://github.com/orchest/orchest https://github.com/orchest/orchest) tldr; a GUI for Argo

adamzegelin 4y ago

Cadence: https://cadenceworkflow.io https://cadenceworkflow.io

kaelin 4y ago

abrazensunset 4y ago

"I want to write my orchestration in Python and I'm comfortable hosting my own compute" -> Prefect (lightweight) or Dagster (heavier but featureful) "My team already knows Airflow and/or I want to pay Astronomer a lot of money" -> Airflow "I love YAML and everything is on k8s anyway" -> Argo "I just want something that works out of the box and don't want to host my own compute" -> Shipyard, maybe Orchest "I want a more flexible, generic workflow engine and don't care about writing orchestration in Python" -> Temporal/Cadence "I am very nostalgic" -> Azkaban, Oozie, Luigi "I love clunky Java solutions to data problems" -> Nifi et al "I like to pay for half-managed solutions and late upgrades to a first-generation technology" -> AWS/GCP hosted Airflow options "I am on AWS and it doesn't need to be complicated" -> AWS Step Functions

herodoturtle 4y ago
This was a really useful comment. Thank you. I know there's a degree of oversimplification going on here, but there's something to be said for having a simple bullet-list breakdown of all the use-cases - alongside the best tool for each use-case. It is servers as a practical starting point in terms of narrowing down the list of tools (of which there are so many), before one proceeds with a deeper dive into the best fitting tool. Would be great if there were a site that did this sort of thing for all the common architectural needs.
corrius 4y ago
+1 to AWS Step Functions, in my last three companies I have built fairly complicated workflows with them and once you get used to them they are very powerful, reliable and cheap. I just wish a little bit more monitoring on top of them but nothing you can not build by yourself.
otabdeveloper4 4y ago
I ended up choosing NodeRed at current job. What does that make me? :)