2 ms·
> teams keep choosing it for building simple DAGs I am part of one such team. We were using Windows Task Scheduler on a windows VM to run jobs, we figured it w
by nomilk 4y ago
> teams keep choosing it for building simple DAGs
I am part of one such team. We were using Windows Task Scheduler on a windows VM to run jobs, we figured it would be a nice idea to (dramatically) modernise and move to airflow, but we grossly underestimated the complexity, learning curve, and surrounding tools it requires. In the end we (data science team) didn't get a single production task up and running. The data engineers had much more success with it thought, probably because they dedicated much more time to it.
Will look forward to trying AWS Step Functions.
- commandlinefan 4y agoI tried installing Airflow locally to just play around with it and make sense of what it's good for and finally gave up after a few days - the install alone is insanely complicated, with lots of tricky hidden dependencies.
- always_left 4y agoDid you try installing with docker? You would just download docker, `docker-compose up --build` and you'll be good to go locally (usually)
- idleprocess 4y agoI can second this. We were up-and-running with Docker on our dev machines in just a few minutes. A native installation involves substantially more setup (Python, databases, Redis and/or Rabbit, etc.). The published docker-compose file will handle all of that for you. We have a very small data engineering team and have been able to move very quickly with Docker and AWS ECS (for orchestrating containers in test and prod environments).
- flowair 4y agodownload the astronomer CLI and it will spin up a local dockerized instance of airflow in about 1 minute tops https://docs.astronomer.io/astro/install-cli https://docs.astronomer.io/astro/install-cli