3 ms·

I've never used Airflow, but used Step Function in AWS to pretty much achieved the same things this article described. I wonder if anybody has used both and wha

I've never used Airflow, but used Step Function in AWS to pretty much achieved the same things this article described. I wonder if anybody has used both and what are the pros and cons between them? Besides the obvious reason of Step Function in AWS so it would work better within AWS ecosystem and Airflow is open source and service/provider agnostic?

rockostrich 5y ago
Yea they look pretty similar. Not sure what the configuration language around step function is, but Airflow is nice because it's just python. There are a few gotchas you encounter, especially around templating, when you first get started but besides that it's pretty low overhead to start doing some pretty complex things since it's just writing python code.
- fredliu 5y ago
  Step Function uses JSON to describe the DAG, or if you use CDK you can define the DAG using any supported language (typescript, python, etc.).
higeorge13 5y ago
A couple points from my end: - I miss the airflow ui to monitor the workflow execution, clear failed tasks, its pre built notifications, emails and so on. - Passing state between step functions is a bit tricky tbh. I found airflow’s way a little more straightforward. - Step functions seem to scale better, especially when you aim to use it for dynamic workflows. The recent update of step functions where you can literally use any aws service api (compared to a couple weeks ago where you were restricted to around 10 and then you only had lambdas) minimises its weaknesses imo.
- fredliu 5y ago
  Ya, the new Step Function service integration feature is awesome, really seals the deal for Step Function being the orchestration tool if you are in AWS.
xkcd-sucks 5y ago
Airflow: Much has been written about good and bad. My bads included poor support for event driven workflows; goods included ease of extensibility. A lot has happened since I stopped using it. AWS stepfunctions: Goods include okay event driven workflows, integration with AWS products, offloading lower level infrastructure to AWS-only people. Bads are everything else - It gives you a set of remarkably and weirdly limited primitives from which you may assemble something like Airflow. If you have any important use case, avoid it like the plague because it is half-baked and limited to a degree that only amazon can accomplish, yet it is capable enough for an initial POC to seem impressive.

rockostrich 5y ago

Yea they look pretty similar. Not sure what the configuration language around step function is, but Airflow is nice because it's just python. There are a few gotchas you encounter, especially around templating, when you first get started but besides that it's pretty low overhead to start doing some pretty complex things since it's just writing python code.

fredliu 5y ago
Step Function uses JSON to describe the DAG, or if you use CDK you can define the DAG using any supported language (typescript, python, etc.).

higeorge13 5y ago

A couple points from my end: - I miss the airflow ui to monitor the workflow execution, clear failed tasks, its pre built notifications, emails and so on. - Passing state between step functions is a bit tricky tbh. I found airflow’s way a little more straightforward. - Step functions seem to scale better, especially when you aim to use it for dynamic workflows. The recent update of step functions where you can literally use any aws service api (compared to a couple weeks ago where you were restricted to around 10 and then you only had lambdas) minimises its weaknesses imo.

fredliu 5y ago
Ya, the new Step Function service integration feature is awesome, really seals the deal for Step Function being the orchestration tool if you are in AWS.

xkcd-sucks 5y ago

Airflow: Much has been written about good and bad. My bads included poor support for event driven workflows; goods included ease of extensibility. A lot has happened since I stopped using it. AWS stepfunctions: Goods include okay event driven workflows, integration with AWS products, offloading lower level infrastructure to AWS-only people. Bads are everything else - It gives you a set of remarkably and weirdly limited primitives from which you may assemble something like Airflow. If you have any important use case, avoid it like the plague because it is half-baked and limited to a degree that only amazon can accomplish, yet it is capable enough for an initial POC to seem impressive.