3 ms·

Hey wanted to give a bit of feedback. We've found Airflow and ECS Fargate to be a great combination for running ETLs. It keeps Airflow small and dumb, and let

Hey wanted to give a bit of feedback. We've found Airflow and ECS Fargate to be a great combination for running ETLs. It keeps Airflow small and dumb, and lets the Fargate containers do the heavy or complicated lifting in language of developer's choice. We'd really appreciate if the ECS Operator could be given a bit of attention: Running a task on FARGATE_SPOT containers is a cheap, convenient option, but it requires passing capacityProviderStrategy in. https://issues.apache.org/jira/browse/AIRFLOW-6604 https://issues.apache.org/jira/browse/AIRFLOW-6604 Also currently the ECSOperator only shows the output logs once the task has finished (which could take hours), it'd be better if the operator could poll the Cloudwatch logs during the run rather than wait for it to finish. --- Congratulations on the release, I'm looking forward to upgrading soon, and trying out the new features and syntax!

dimberman 6y ago
Thank you for the feedback! I'm gonna pass that on to some AWS experts in the community. One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released separately from Airflow itself. So you won't need to upgrade airflow to get improved AWS operators unless there is a breaking change.
- hbcondo714 6y ago
  Ditto, running Airflow on AWS ECS Fargate serverless. We did this prior to AWS announcing their Managed Workflows for Apache Airflow[1]. Do you know if and when AWS will be making Airflow 2.0 available in their managed service? [1] https://aws.amazon.com/managed-workflows-for-apache-airflow/ https://aws.amazon.com/managed-workflows-for-apache-airflow/
  - politelemon 6y ago
    Have you tried Managed Workflows for Apache Airflow? I was curious about it but the pricing page scared me off, the smallest which runs 50 DAGs is about $0.49/hr! I couldn't understand why the pricing was that way.
  - dimberman 6y ago
    I'm sure they will, I'm not sure on their timeline though.
deleted 6y ago
[deleted]

dimberman 6y ago

Thank you for the feedback! I'm gonna pass that on to some AWS experts in the community. One really nice feature of 2.0 is now the "providers (hooks, operators, etc.) are released separately from Airflow itself. So you won't need to upgrade airflow to get improved AWS operators unless there is a breaking change.

hbcondo714 6y ago
Ditto, running Airflow on AWS ECS Fargate serverless. We did this prior to AWS announcing their Managed Workflows for Apache Airflow[1]. Do you know if and when AWS will be making Airflow 2.0 available in their managed service? [1] https://aws.amazon.com/managed-workflows-for-apache-airflow/ https://aws.amazon.com/managed-workflows-for-apache-airflow/
- politelemon 6y ago
  Have you tried Managed Workflows for Apache Airflow? I was curious about it but the pricing page scared me off, the smallest which runs 50 DAGs is about $0.49/hr! I couldn't understand why the pricing was that way.
- dimberman 6y ago
  I'm sure they will, I'm not sure on their timeline though.

deleted 6y ago

[deleted]