4 ms·
Thanks for your comment! Ditto last time I ran Airflow locally it took like 5 Docker containers. Then I forgot about the project and for a while was furious at
by kbd 4y ago
Thanks for your comment! Ditto last time I ran Airflow locally it took like 5 Docker containers. Then I forgot about the project and for a while was furious at Docker for randomly taking 100% CPU. Then I realized it was because of the Airflow containers that would restart along with Docker. I didn't get much further with Airflow.
Dagster, on the other hand, seems to let you scale from using it locally as a library all the way to running on ECS/K8s etc. Along with that there's unfortunately a ton of complexity in setting it up but that's not Dagster's fault and it seems like Dagster works once you get it set up. Agree about it being young and there being some rough spots but it's got lots of good ideas. We were nearly done setting it up but got pulled off onto more urgent things, so I haven't run it in production yet. I'm glad to hear it worked well for you!