5 ms·

We tested it, but the performancd was bad. We needed hundred workflows with few hundred taks each, and Airflow would just topple over daily. We ended up with p

by glogla 8y ago

We tested it, but the performancd was bad. We needed hundred workflows with few hundred taks each, and Airflow would just topple over daily. We ended up with proprietary tool from Teradata thats basically Airflow written in perl - but it can handle all the work. Other than scalability, Airflow is pretty nice.

tpaschalis 8y ago
It sounds weird that you had these problems. Silly question, but might it have been a database optimization issue?
vgy7ujm 8y ago
Perl saved the day again.
caravel 8y ago
[full disclosure, I'm the creator of Airflow] Many environments run tens of thousands of concurrent tasks, and hundreds of thousands of tasks daily. The list of companies using Airflow speaks for itself https://github.com/apache/incubator-superset#who-uses-apache-superset-incubating https://github.com/apache/incubator-superset#who-uses-apache... But hey, it's like anything, you have to do a bit of work to get distributed systems to run at scale. There are now hosted solutions to help with that (Google Cloud Composer and Astronomer.io)
- tedmiston 8y ago
  https://github.com/apache/incubator-airflow#who-uses-airflow https://github.com/apache/incubator-airflow#who-uses-airflow :)
tedmiston 8y ago
I run an Airflow instance that does millions of tasks per month across dozens of DAGs. There's some performance tuning involved in the configuration file and of course you need the underlying resources available but Airflow has scaled to this level well for us. If you are able to reproduce and can post to the dev mailing list, we are happy to help... especially so if it gets you off of a proprietary tool written in Perl ;). 1.10 was just released and adds a ton of commits. I'd really encourage you to give Airflow another shot if you have the time.
- vgy7ujm 8y ago
  Millions per MONTH is just ridiculous. Stay with what works, revisiting this slow Python tool will be a cost and time sink. A mistake companies let engineers with millennial complex do all too often.
rywalker 8y ago
My guess is that you had an underpowered database instance backing Airflow.

by glogla 8y ago

tpaschalis 8y ago

It sounds weird that you had these problems. Silly question, but might it have been a database optimization issue?

vgy7ujm 8y ago

Perl saved the day again.

caravel 8y ago

[full disclosure, I'm the creator of Airflow] Many environments run tens of thousands of concurrent tasks, and hundreds of thousands of tasks daily. The list of companies using Airflow speaks for itself https://github.com/apache/incubator-superset#who-uses-apache-superset-incubating https://github.com/apache/incubator-superset#who-uses-apache... But hey, it's like anything, you have to do a bit of work to get distributed systems to run at scale. There are now hosted solutions to help with that (Google Cloud Composer and Astronomer.io)

tedmiston 8y ago
https://github.com/apache/incubator-airflow#who-uses-airflow https://github.com/apache/incubator-airflow#who-uses-airflow :)

tedmiston 8y ago

I run an Airflow instance that does millions of tasks per month across dozens of DAGs. There's some performance tuning involved in the configuration file and of course you need the underlying resources available but Airflow has scaled to this level well for us. If you are able to reproduce and can post to the dev mailing list, we are happy to help... especially so if it gets you off of a proprietary tool written in Perl ;). 1.10 was just released and adds a ton of commits. I'd really encourage you to give Airflow another shot if you have the time.

vgy7ujm 8y ago
Millions per MONTH is just ridiculous. Stay with what works, revisiting this slow Python tool will be a cost and time sink. A mistake companies let engineers with millennial complex do all too often.

rywalker 8y ago

My guess is that you had an underpowered database instance backing Airflow.