3 ms·

Color me very interested about footguns in Airflow as well. We're currently considering it as a self-hosted process engine to automate technical business proce

by tetha 3y ago

Color me very interested about footguns in Airflow as well. We're currently considering it as a self-hosted process engine to automate technical business processes and to coordinate a few automation systems like jenkins. Crap like, trigger a database restore via system A, wait until complete, update tickets, trigger some data migration from that database via System B, update tickets. Maybe bounce things back to human operators on errors and wait for fixing / clarification. Trigger a bunch of deployments in parallel and report on success. Systems in this space are either (a) huge, hugely expensive enterprise applications designed to consume all business processes, which is a bit overkill for our current needs (Camunda, SAP, Stackstorm, ...), (b) overfitted onto a very specific data analysis setup (aka: if you don't have hadoop, don't touch it) or (c) overly simplistic and offering no real benefit beyond investing into self-hatred, guiness and making jenkins work. Airflow seemed like a bit of a decent middle ground there for workflows a bit beyond what you can sensibly do via jenkins jobs.

abrazensunset 3y ago
Please don't do this. If you want an Airflow-ish approach without punishing your future self, pick Prefect. Otherwise go with Temporal. Above all do not adopt Airflow for the use cases you describe in 2023
- tetha 3y ago
  Prefect seems like a really good suggestion, thank you. We're pretty much committing to python as our language of choice in the infra-layer. Most of the team is sent onto courses over the next month, too. So I have a whole lot of python scripts popping up over the infrastructure. And this approach of slapping some @task and some @flow onto scripts or helper functions seems to work really well with what the team is doing. It took me like 30 - 40 minutes to convert one of those scripts into what seems a decently fine workflow. very intrigued.
slotrans 3y ago
Jenkins is honestly Really Good as a scheduling system. The best, frankly. If you can combine it with a library- or tool-style DAG system (vs. server-style like Airflow), like make or Luigi or NextFlow or even Step Functions, that is a great sweet spot.
- dharmab 3y ago
  In my experience Jenkins is like shell scripting- in the hands of hands of someone who understands its strengths and weaknesses and is very disciplined in how it is used and maintained, it's both performant and flexible. If you follow the path of least resistance it becomes a mess.
- patrick451 3y ago
  I agree. For some reason, Airflow really sucks at running something on a cron-like timer. I can't remember the all issues I had with it exactly (this was several jobs ago), but getting it to run a job at the same time everyday was a nightmare. One issue I do recall was that it treated the first run differently, basically ignoring your defined schedule. And I think "first run" meant every time Airflow was restarted. So if Bad Things happened if that job ran the wrong time of day, you had to add extra logic into the job itself to abort if it was being called at the wrong time. How they managed to make this so difficult is mind boggling. By contrast, Jenkins will reliably execute a job on a timer without issues.
pani5ue 3y ago
Use makefiles for as long as you can.. If your DAGs get too complicated for makefiles, it's time to rethink and restructure your DAGs

by tetha 3y ago

abrazensunset 3y ago

Please don't do this. If you want an Airflow-ish approach without punishing your future self, pick Prefect. Otherwise go with Temporal. Above all do not adopt Airflow for the use cases you describe in 2023

tetha 3y ago
Prefect seems like a really good suggestion, thank you. We're pretty much committing to python as our language of choice in the infra-layer. Most of the team is sent onto courses over the next month, too. So I have a whole lot of python scripts popping up over the infrastructure. And this approach of slapping some @task and some @flow onto scripts or helper functions seems to work really well with what the team is doing. It took me like 30 - 40 minutes to convert one of those scripts into what seems a decently fine workflow. very intrigued.

slotrans 3y ago

Jenkins is honestly Really Good as a scheduling system. The best, frankly. If you can combine it with a library- or tool-style DAG system (vs. server-style like Airflow), like make or Luigi or NextFlow or even Step Functions, that is a great sweet spot.

dharmab 3y ago
In my experience Jenkins is like shell scripting- in the hands of hands of someone who understands its strengths and weaknesses and is very disciplined in how it is used and maintained, it's both performant and flexible. If you follow the path of least resistance it becomes a mess.
patrick451 3y ago
I agree. For some reason, Airflow really sucks at running something on a cron-like timer. I can't remember the all issues I had with it exactly (this was several jobs ago), but getting it to run a job at the same time everyday was a nightmare. One issue I do recall was that it treated the first run differently, basically ignoring your defined schedule. And I think "first run" meant every time Airflow was restarted. So if Bad Things happened if that job ran the wrong time of day, you had to add extra logic into the job itself to abort if it was being called at the wrong time. How they managed to make this so difficult is mind boggling. By contrast, Jenkins will reliably execute a job on a timer without issues.

pani5ue 3y ago

Use makefiles for as long as you can.. If your DAGs get too complicated for makefiles, it's time to rethink and restructure your DAGs